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1.  Introduction 


Robots  and  other  unmanned  systems  have  been  increasingly  used  for  various  tasks,  including 
transportation  safety,  space  exploration,  and  military  operations  (Barnes  and  Evans,  2010;  Jones 
and  Schmidlin,  2011;  Murphy  and  Burke,  2010).  In  the  U.S.  military,  there  is  a  trend  to 
implement  robotic  systems  that  can  perform  some  of  the  functions  on  the  battlefield  currently 
conducted  by  Soldiers  (e.g.,  casualty  extraction,  explosive  detection  and  disposals, 
reconnaissance  and  surveillance,  supply  transportation,  building  clearing,  and  firefighting,  etc.; 
Bames  and  Evans,  2010;  Greenemeier,  2010;  Osbom,  2011;  Purdy,  2008;  Singer,  2010).  The 
same  trend  is  manifesting  itself  in  the  civilian  sector  with  the  advent  of  the  Goggle  driverless  car, 
autonomous  farm  equipment,  and  unmanned  commercial  aircraft  (Mosher,  2012). 

The  inexorable  trend  towards  greater  societal  reliance  on  unmanned  systems  begs  the  question  of 
control.  Based  on  numerous  real-world  experiences  in  robotics  operations  during  emergencies 
(e.g..  World  Trade  Center,  Hurricane  Katrina,  and  Utah  Mine  disaster),  Murphy  and  Burke 
(2010)  suggest  that,  with  current  technology,  a  two  humans  to  one  robot  is  a  minimum  ratio 
because  of  the  safety  issues  involved.  The  ratio  increases  depending  on  the  complexity  of  the 
environment  and  the  degree  of  automaton.  A  higher  ratio  of  humans  is  particularly  important  for 
combat  environments,  where  fratricide  or  civilian  casualties  are  possible.  The  ratio  is  high 
because  of  the  limited  situation  awareness  (SA)  that  the  operator  has  when  viewing  the  world 
from  a  robot’s  portal —  not  because  a  robot  cannot  be  automated  to  navigate  in  difficult  terrain. 

However,  this  high  ratio  will  not  be  sustainable  if  large  numbers  of  autonomous  systems  are 
deployed  for  civilian  and  military  applications.  This  will  require  supervision  of  not  only  single 
systems  but  also  larger  scale  systems  while  maintaining  an  economically  feasible  ratio  of 
humans  to  autonomous  systems  (Chen  et  al.,  2011;  Cummings  et  al.,  2010).  Increases  in  the 
sophistication  and  degree  of  autonomy  will  be  necessary  but  not  sufficient  to  maintain  large  scale 
unmanned  systems,  due  to  the  human  performance  issues  associated  with  increased  autonomy: 
tunnel  vision,  misuse  and  disuse  of  automated  systems,  complacency,  and  degraded  SA  (Chen 
et  al.,  2011;  Lee,  2008;  Parasuraman  and  Riley,  1997).  More  germane  to  our  current  discussion, 
increases  in  autonomy  may  not  overcome  the  human’s  span  of  apprehension  limits  around  7  (±2) 
related  to  monitoring  multiple  systems  at  the  same  time  (Lewis  et  al.,  2010;  Miller,  1956;  Wang 
et  al.,  2009). 

In  recent  years,  there  have  been  several  research  efforts  on  developing  intelligent  systems  that 
can  assist  the  human  operators  to  manage  multiple  robots  in  military  tasking  environments  (Chen 
and  Bames,  2012a;  Fem  and  Shively,  2009;  Miller  and  Parasuraman,  2007).  The  purpose  of  this 
report  is  to  review  the  literature  on  intelligent  systems  and  discuss  their  efficacy  in  relation  to 
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•  efficient  human  supervision  of  multiple  robots. 

•  appropriate  human  trust  in  the  automated  systems. 

•  maintenance  of  human  operator’s  SA. 

•  individual  differences  in  human-agent  (H-A)  interaction. 

•  retention  of  human  decision  authority. 

In  the  first  part  of  this  report  (Sections  2  and  3),  we  review  a  number  of  topics  from  flexible 
automation  to  autonomous  agents  (e.g.,  mixed-initiative  H-A  teams,  cognitive  architectures)  and 
point  out  the  advantages  and  disadvantages  of  the  various  approaches.  In  particular,  we  will 
address  the  advantages  of  mixed  initiative  models  to  leverage  both  human  and  agent  decision 
strengths.  To  illustrate  one  such  approach,  we  discuss  briefly  some  of  our  own  research  for  a 
specific  hybrid  approach  (RoboLeader)  and  present  some  of  our  major  findings.  In  the  second 
part  of  this  report  (Sections  4-6),  we  discuss  in  detail  two  key  human  performance  issues  (trust 
and  SA)  related  to  H-A  teaming  for  multi-robot  control.  After  reviewing  research  findings  on 
these  two  issues,  we  also  present  some  promising  user  interface  design  solutions  to  address  these 
issues.  Also  in  the  second  part  of  the  report,  we  discuss  briefly  some  key  individual  differences 
factors  (operator  spatial  ability,  attentional  control  ability,  and  gaming  experience)  that  may 
impact  H-A  teaming  in  the  context  of  robotics  control.  At  the  end  of  the  report  (Section  7),  we 
briefly  summarize  some  design  guidelines  based  on  our  own  and  reviewed  research. 


2.  H-A  Teaming 


Before  we  delve  into  the  discussion  of  all  the  issues  related  to  H-A  teaming,  a  few  notes  on  the 
terminology  seem  to  be  in  order.  While  there  are  numerous  definitions  of  “agent”  (including 
intelligent  agent,  software  agent,  etc.),  it  is  commonly  accepted  that  an  agent  should  possess  the 
following  characteristics:  autonomy,  observation  of  the  environment  (through  some  forms  of 
sensors),  action  upon  an  environment  (through  some  forms  of  actuators),  and  direction  of  its 
activity  towards  achieving  certain  goals  (Russell  and  Norvig,  2009).  The  complexity  of  these 
systems  varies  greatly  and  ranges  from  a  simple  “reflex  agent”  (e.g.,  a  thermostat)  to  an  agent 
that  can  learn  and  evolve.  While  robots,  by  definition  are  agents,  agents  do  not  necessarily  have 
physical  embodiment.  In  the  context  of  our  current  discussion,  we  will  use  the  term  agent  to 
refer  to  a  broader  class  of  intelligent  systems  (with  or  without  physical  embodiment)  that  are 
consistent  with  the  definition  given  above.  We  will  use  the  term  robots  when  the  physical 
embodiment  component  is  important  in  the  discussion. 
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2.1  Characteristics  of  H-A  Teams 

Human-agent  teams  assume  a  relationship  that  goes  beyond  a  human  controlling  or  supervising 
agents  (Inagaki,  2008).  Teams  are  more  than  intelligent  entities  working  on  a  common  problem. 
Teams  coordinate  (share  knowledge  and  depend  on  each  other’s  output)  and  collaborate  (work 
together  on  common  functions)  (Bradshaw  et  al.,  201 1).  They  share  a  common  knowledge 
framework  but  it  is  not  necessary  that  all  team  members  have  the  same  information;  it  is  efficient 
for  some  team  members  to  have  specialized  knowledge  as  long  as  they  coordinate  with  the  rest 
of  the  team  (Cooke  et  al.,  2000).  Coordination  requires  communication  among  team  members  as 
well  as  shared  knowledge.  Effective  teams  can  communicate  implicitly  as  well  as  explicitly. 
Implicit  communication  depends  on  individuals  knowing  their  role  and  how  it  intersects  with 
other  team  members’  roles  to  obtain  tasking  objectives.  It  is  important  to  note  that  too  much 
explicit  communication  can  degrade  team  effectiveness  as  much  as  too  little  communication 
(Cannon-Bowers  et  al.,  1998;  Nikolaidis  and  Shah,  2012;  Steinberg,  2012).  Thus  a  H-A  teaming 
relationship  assumes: 

•  common  knowledge 

•  shared  understanding  of  objectives 

•  two-way  communication 

There  are  a  number  of  ways  to  communicate  with  an  agent.  For  example,  communication  can  be 
implemented  with  graphical  displays  that  can  be  updated  by  both  the  human  and  the  agent  while 
keeping  both  parties  informed  of  the  current  state  of  the  tasking  objectives  (Chen  and  Barnes, 
2012b).  In  order  to  make  H-A  interactions  as  natural  as  human-human  interactions,  research 
efforts  have  been  focusing  on  ways  to  support  natural  language  communications  between 
humans  and  agents  (Tellex  et  al.,  2011).  Effective  natural  language  communications  require  not 
only  disambiguating  the  syntax  and  semantics  of  the  utterance  but  also  its  pragmatics  relating  to 
the  tasking  environment  and  the  intention  of  the  speaker  (Jurafsky  and  Martin,  2009).  The 
current  status  of  human-robot  communications  typically  entails  humans  giving  the  robot  a  direct 
command  and  the  robot  trying  to  understand  the  command  in  terms  of  its  physical  environment. 
Even  with  relatively  constrained  environments,  agents  can  be  confused  by  ambiguous  commands 
that  humans  could  decipher  easily  (Cassenti  et  al.,  in  review;  Tellex  et  al.,  201 1).  The  additions 
of  gestures,  gaze-facilitated  language  processing,  and  better  training  protocols  are  enhancing  the 
ability  of  humans  and  robots  to  engage  in  more  natural  two-way  communications,  but  natural 
language  processing  (NLP)  is  still  relatively  unsophisticated  compared  to  human-to-human 
dialogue  (Camak  and  Thomaz,  2012). 

However,  the  most  important  obstacle  to  effective  H-A  teaming  is  not  NLP  limitations  but  rather 
a  shared  understanding  of  the  implicit  goals  inherent  in  complex  environments.  Trying  to 
enumerate  all  possible  intent  situations  is  not  practical.  For  example,  telling  a  robot  to  go  to  the 
back  door  of  a  building  and  check  for  improvised  explosive  devices  (IEDs)  require  not  only 


3 


sophisticated  navigation  and  surveillance  capabilities  but  also  intent  inferencing.  For  human 
teams,  it  would  require  a  deeper  understanding  of  the  supervisor’s  intent  including  checking  for 
IEDs  or  anything  suspicious  while  moving  towards  the  objective.  Developing  intent  inferencing 
software  has  a  long  history;  early  attempts  used  pilot  actions  such  as  button  pushes  to  infer  the 
pilot’s  objectives  (Rouse  et  al.,  1990).  These  attempts  proved  to  be  too  brittle  and  to  have 
limited  utility  for  real-world  situations  (Banks,  2002).  Recently,  more  sophisticated  software 
using  intent-driven  behavioral  modeling  have  had  some  success  in  predicting  human  behaviors 
and  intentions;  however  the  solutions  are  more  generic  than  would  be  practical  for  a  typical 
human-agent  problem  space  (Santos  et  al.,  2012).  Thus  in  the  foreseeable  future,  it  does  not 
seem  likely  that  an  agent  can  communicate  with  humans  as  peers  in  any  but  the  most  constrained 
environment.  This  is  due  to  unique  human  characteristics  (discussed  below)  as  well  as 
communication  limitations.  The  human  role  as  part  of  the  team  equation  is  explored  in  the  next 
sections. 

2.2  The  Human  Role — Strengths  and  Weaknesses  of  Human  Reasoning 

Successful  agent  technologies  take  advantage  of  the  differences  between  human  and  agent 
strengths,  as  human  reasoning  has  very  different  characteristics  than  algorithmic  reasoning 
(Bradshaw  et  al.,  2011).  Human-agent  teams  seem  to  be  particularly  effective  for  open-ended 
missions  in  which  not  all  events  can  be  pre-programmed  (e.g.,  most  combat  situations).  Agents 
can  perform  specialized  functions  but  authority  resides  with  a  human  supervisor  for  safety  and 
tactical  reasons  (Barnes  and  Evans,  2010;  Woods  et  al.,  2004).  This  section  reviews  some  of  the 
most  relevant  human  strengths  and  weakness  in  the  context  of  H-A  interaction  for  supervisory 
control  tasks. 

2.2.1  Human  Rational  Process 

Johnson-Laird  (2010)  points  out  that  humans  are  rational  but  do  not  use  formal  logic  in  everyday 
decision  making.  They  tend  to  structure  problems  by  focusing  on  only  some  of  the  possible 
logical  implications,  permitting  humans  to  rapidly  but  sometimes  erroneously  solve  real-world 
problems.  Kahneman  (2003)  argues  that  many  human  decision  processes  are  examples  of 
bounded  rationality.  These  strategies  lead  to  quick  decisions  that  have  their  evolutionary 
pedigree  in  the  ability  of  humans  to  recognize  dangerous  situations  without  extensive  processing. 
Klein  (2008)  points  out  that  human  intuition  is  not  an  inexplicable  process  but  rather  it  involves 
matching  multiple  memory  traces  with  current  cues  that  alert  humans  to  possible  incongruities  in 
the  environment.  Experts,  in  particular,  are  able  to  pick  up  on  cues  that  allow  them  to 
circumvent  detailed  analysis  (Klein  and  Jarosz,  2011).  A  good  example  involves  Klein’s 
interviews  with  expert  fire  fighters.  The  experts  reported  that  considering  various  options  during 
a  fire  could  be  fatal  because  of  the  time  constraints  involved.  Klein  posits  that  the  Recognition 
Primed  Decisions  (RPD)  process  is  frequently  involved  in  such  situations.  A  good  solution 
seems  to  “pop-up,”  allowing  humans  to  rapidly  discover  a  safe  course  of  action.  This  is  not  a 
magical  process;  RPDs  are  based  on  experts  being  able  to  abstract  years  of  experience  to  serve 
them  in  times  of  danger.  Finally,  humans  are  able  to  understand  not  only  the  literal  objective  of 
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a  stated  mission  but  also  its  intent  and  even  its  larger  implications.  Thus,  an  experienced 
commander  may  seize  an  unoccupied  hill  because  he  or  she  realizes  its  greater  tactical  value 
compared  to  the  original  objective.  The  decision  may  also  be  influenced  by  strategic  objectives 
such  as  reducing  civilian  casualties.  There  is  no  guarantee  that  these  decisions  will  be  correct, 
but  such  flexible  decision-making  permits  humans  to  arrive  at  nuanced  solutions  not  simply  rule- 
based  ones.  Recently,  the  RPD  framework  has  been  applied  to  the  development  of  intelligent 
agents  that  can  collaborate  with  humans  in  time-stressed  situations,  and  section  3.1  will  review 
some  of  the  efforts  in  this  area  (Fan  et  al.,  2010). 

2.2.2  The  Aspects  of  Human  Emotion 

Until  recently,  human  emotional  responses  were  neglected  in  most  studies  of  human  cognitive 
processes  because  emotions  were  considered  a  detriment  or  irrelevant  to  problem  solving. 
However,  more  recent  research  has  pointed  out  its  importance  to  effective  decision  making 
(Damasio,  2010;  Lee  and  See,  2004;  Ramachandran,  2011).  Emotions  sensitize  humans  to  subtle 
cues  in  the  environment,  creating  an  awareness  of  possible  dangerous  indicators  that  would  be 
difficult  to  recreate  artificially.  There  is  a  direct  loop  between  possibly  dangerous  external 
stimuli  and  the  human  brain’s  emotional  centers  (amygdala)  allowing  for  rapid  although 
relatively  unanalyzed  responses  to  the  stimuli.  This  would  explain  a  Soldier’s  heightened 
sensitivity  to  changes  in  the  immediate  environment  before  executive  functions  in  the  neo-cortex 
have  fully  evaluated  the  situation  (Goleman,  1995).  Damasio  (2010)  also  points  out  emotional 
centers  in  the  brain  have  numerous  interconnections  to  the  executive  centers  in  the  frontal  lobe, 
suggesting  the  importance  of  human  emotional  process  in  more  reasoned  decision  making. 
Humans  are  not  passive  processors;  particularly  when  their  emotions  are  engaged,  they  are 
motivated  to  acquire  new  skills  and  to  develop  new  approaches  to  problem  solving  in  response  to 
environmental  changes.  Another  important  advantage  to  the  relationship  between  human 
cognition  and  emotion  is  the  ability  to  empathize  with  other  humans — making  other  humans’ 
intentions  more  transparent.  Recently,  researchers  have  discovered  mirror  cells  that  are  fired  in 
relation  to  actions  of  another  human  suggesting  that  humans  have  specialized  neurons  for 
predicting  the  intentions  of  other  humans  (Ramachandran,  2011).  In  summary,  human  emotional 
capabilities  are  important  because  they  are  intimately  related  to  human  executive  functions, 
sensitize  humans  to  dangers  and  to  opportunities  in  their  environment,  and  are  important  in 
assisting  humans  to  understand  the  intent  of  other  humans  by  their  body  language  and  other 
subconsciously  processed  cues  (Ramachandran,  201 1). 

2.2.3  Heuristics  and  Biases 

The  other  side  of  the  coin  is  the  tendency  of  humans  to  use  heuristics  to  solve  real-world 
problems,  especially  in  a  time-constrained  environment.  Heuristic  approaches  lead  to  biases, 
which,  in  turn,  lead  to  predictable  errors  (Barnes  et  al.,  2011;  Kahneman,  2003).  For  example, 
the  availability  heuristic  depends  on  the  ease  of  imagining  an  instance  of  an  event  rather  than  an 
understanding  of  the  statistical  likelihood  of  an  event  occurring  (Tversky  and  Kahneman,  1974). 
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Parasuraman  and  Manzey  (2010)  examined  the  role  of  the  “automation”  bias  for  decision  support 
systems,  particularly  its  implication  for  complacency  errors.  Both  phenomena  are  most  salient  in 
multitasking  environments,  for  support  systems  that  are  highly  accurate,  and  in  situations  where 
the  outcomes  are  consistent  (Rovira  et  al.,  2007).  One  counterargument  is  that  there  is  no  bias  or 
error  involved — the  participants  were  simply  sampling  information  from  different  tasks  in  an 
optimal  fashion;  they  were  not  ignoring  the  automated  task,  rather  they  were  under- sampling 
automation  to  reflect  the  relative  payoffs  of  sampling  strategies  in  a  high  taskload  situation 
(Moray,  2003).  Parasuraman  and  Manzey  (2010)  detailed  a  series  of  studies  that  indicated  the 
dynamics  of  the  automation  bias  (i.e.,  complacency  effects)  were  due  to  attentional  deficits  rather 
than  sampling  strategies  or  decision  biases  as  described  in  Tversky  and  Kahneman  (1974).  The 
data  of  Chen  and  her  colleagues  (Chen  and  Barnes,  2012a;  Chen  and  Terrence,  2009)  supported 
this  hypothesis  that  automation  bias  is  related  to  the  human  operator’s  attentional  control  abilities 
(see  section  6  on  Individual  Differences).  In  Parasuraman  and  Manzey  (2010),  the  tasks 
involved  supervisory  control  of  five  space  station  subsystems  and  automated  fault  diagnostic 
systems  that  provided  fault  diagnosis  and  suggested  remedial  actions.  The  aid  suggested 
erroneous  actions  in  20%  of  the  trials  but  allowed  the  participants  (engineering  students)  access 
to  the  correct  diagnostic  information.  The  sampling  behavior  was  suboptimal,  but  the  most 
striking  result  was  the  type  of  errors  the  participants  committed.  Even  when  participants 
accessed  correct  diagnostic  information,  they  tended  to  follow  the  (erroneous)  aid-suggested 
course  of  action.  Some  of  the  participants  even  misreported  accessing  information  that 
supported  the  incorrect  remedial  action  — replicating  the  “phantom  memory”  effects  found  in 
previous  automation  bias  research  (Bahner  et  al.,  2008).  In  a  recent  study,  Manzey  et  al.,  (2012), 
further  demonstrated  that  the  human  complacency  behaviors  are  closely  linked  to  attention  (i.e., 
insufficient  attention  and  incomplete  verification,  and  inattentive  processing  of  contradictory 
information).  This  is  a  crucial  point  that  will  be  elucidated  further  in  the  Trust  section  (see  also 
Chen  and  Barnes,  2012a). 

2.3  The  Agent  Role — Strengths  and  Weaknesses  of  Machine  Processing 

Human  factors  researchers  have  been  studying  the  relationship  between  machine  and  human  skill 
sets  since  shortly  after  World  War  II — humans  are  more  effective  at  inductive  processes, 
judgments,  and  flexibility,  whereas  machines  are  better  at  repetitive  actions,  computation,  and 
rapidity  (Fitts,  1951).  Hoffman  et  al.,  (2002)  argues  that  humans  and  intelligent  agent  systems 
work  most  effectively  in  concert,  not  separated  into  skill  domains.  Optimization  methods  and 
algorithmic  solutions  are  relevant  if  informed  by  human  context  and  situational  understanding, 
whereas  humans  perform  more  efficiently  if  they  are  supported  by  computational  and  symbolic 
processing.  In  both  instances,  it  is  the  interaction  of  the  human  and  the  intelligent  system  that  is 
important,  not  the  singular  advantage  of  one  or  the  other  (Bradshaw  et  al.,  201 1).  Kahneman  and 
Klein  (2009)  discussed  the  relative  merits  of  human  decision  making  and  mathematical 
approaches  such  as  regression  equations.  Kahneman  and  Klein  pointed  to  early  studies  that 
showed  human  experts  were  often  outperformed  by  statistical  models  for  making  predictions 
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(e.g.,  clinical  judgments,  college  admissions,  patient  success,  officer  retentions,  etc.)  even  when 
the  experts  shared  the  same  data  sources.  Experts  tended  to  be  overly  influenced  by  personality 
factors  compared  to  objective  data  trends.  Thus  for  situations  that  were  fairly  stable,  algorithms 
based  on  reliable  data  sources  often  outperforms  humans.  However,  Klein  pointed  out  that  these 
approaches  were  not  useful  for  rapidly  changing  situations  or  anomalous  events  requiring 
immediate  decisions  based  on  accumulated  experience.  Machine  solutions  inevitably  improve  as 
computer  technology  improves.  Big  Blue,  the  IBM  super  computer,  now  outperforms  the  best 
chess  experts,  and  machine  approaches  can  be  expected  to  improve  in  all  areas  where 
optimization  approaches  are  feasible;  limits  that  are  constantly  being  redefined  as  research  into 
these  areas  continues  (Russell  and  Norvig,  2009).  An  example  more  relevant  to  human-robot 
interaction  was  provided  by  Cummings  et  al.,  (2010) — it  took  the  human  operators  (who  were  all 
experts)  several  minutes  to  solve  a  simple  path  planning  problem  but  it  only  took  the  intelligent 
agent  seconds  to  perform  the  same  task.  Nevertheless,  Woods  et  al.,  (2004)  point  out  that  final 
authority  for  agent-human  decisions  is  a  human  prerogative,  because  only  the  human  can  be  a 
stakeholder  and  only  the  human  can  be  held  responsible  for  the  outcome. 


3.  Agents  for  Human-Robot  Teaming 


In  artificial  intelligence,  an  intelligent  agent  is  typically  defined  as  “an  autonomous  entity  which 
observes  and  acts  upon  an  environment  and  directs  its  activity  towards  achieving  goals”  (p.  34; 
Russell  and  Norvig,  2009).  This  definition  covers  a  variety  of  possible  uses  for  intelligent 
agents,  from  swarms  with  individual  agents  of  limited  intelligence  that  evince  sophisticated 
behaviors  holistically,  to  agents  that  respond  to  particular  tasks  in  a  manner  that  emulates  human 
intelligence.  The  necessity  for  more  powerful  intelligent  agents  that  can  interact  with  human 
operators  in  increasingly  sophisticated  ways  will  require  that  current  capabilities  be  augmented 
with  techniques  and  technologies  that  facilitate  effective  H-A  team  interactions  (Bradshaw  et  al., 
2011;  Green  et  al.,  2008).  This  section  will  first  briefly  review  cognitive  architectures  relevant  to 
human-robot  interaction  (HRI),  and  then  we  will  discuss  several  types  of  agents  that  have  been 
used  in  HRI  tasks,  including  the  RoboLeader  agent  that  was  developed  in  our  own  laboratory. 
Table  1  presents  a  summary  of  the  technologies  discussed  in  this  section  as  well  as  strengths  and 
weaknesses  of  each  approach. 
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Table  1.  Intelligent  systems  for  multi-robot  control. 


Intelligent 

Systems 

Potential  Strengths 

Potential  Weaknesses 

Key  References 

Cognitive 

Architectures 

Use  of  machine  learning,  neural 
nets,  and  natural  language 
processing  algorithms  that  are 
compatible  with  human  knowledge 
structures. 

Less  efficiency  due  to  complexity 
inherent  in  cognitive  approaches. 

Laird  et  al.,  201 1; 
Kelley,  2006; 

Kurup  et  al.,  2011 

Teaming 

Agents 

Supporting  optimal  task-allocation 
planning  and  flexible  plan- 
execution;  allowing  graceful  plan- 
degradations;  facilitating  proactive 
information  distribution. 

Human- Agent  team  performance 
more  sensitive  to  the  changes  of 
tasking  complexities  (i.e.,  degrading 
faster)  than  Human-Human  teams. 

Fan  et  al.,  2010; 
Korsah  et  al., 

2012 

Hierarchical 

agent 

Greater  flexibility  in  adjusting  to 
task  complexity  (dividing  task 
complexity  between  senior  agents 
and  specialized  agents). 

Communications  among  agents; 
difficulty  in  dealing  with  truly  novel 
situations  (algorithms  too 
cumbersome  when  dealing  with 
complex  hierarchies). 

Hou  etal.,  2011; 
Strenzke  et  al., 

2011 

Adaptive 

Automation 

Balance  between  reducing 
workload  for  manual  control  and 
reducing  SA  loss  for  automated 
tasks;  reducing  the  task 
management  load;  particularly 
useful  in  emergency  situations. 

Harmful  effects  associated  with 
sudden  changes  in  task  state 
(triggered  inappropriately);  usurping 
delegation  authority  from  the  human. 

Parasuraman  et 
al.,  2009; 
Steinhauser,  et  al., 
2009 

Adjustable 

Automation 

Human  initiating  adaptivity  of  the 
joint  human-system  task. 

Increased  workload  associated  with 
selecting  appropriate  autonomy 
levels,  especially  under  emergency 
situations. 

Miller  and 
Parasuraman, 

2007 

Mixed- 

Initiative 

Automation 

Collaborative  decision  making 
between  a  human  and  an  intelligent 
system. 

Communication  mechanisms  between 
the  agents  and  the  humans  to  express 
their  intents;  mixed -initiative 
system’s  ability  to  respond  to 
developments  in  the  tasking 
environment. 

Chen  and  Barnes, 
2012a;  Hardin 
and  Goodrich, 

2007;  Tecuci 
et  al.,  2007 

3.1  Cognitive  Architectures 

Cognitive  architecture  such  as  Adaptive  Control  of  Thought-Rational  (ACT-R)  and  Symbolic 
Cognitive  Architecture  (SOAR)  were  developed  to  mimic  human  intelligence  using  top-level 
symbolic  systems  based  on  production  systems  (i.e.,  if-then  rules)  as  well  as  sub-symbolic 
processes  (Kelley,  2006;  Laird  et  al.,  2011).  The  number  of  cognitive  architectures  has  increased 
greatly  in  the  last  30  years  principally  by  coupling  cognitive  approaches  with  other  artificial 
intelligence  approaches.  For  example,  Kurup  et  al.,  (2011)  combined  perceptual  algorithms  used 
in  robotics  with  ACT-R  to  predict  pedestrian  behavior  in  urban  street  scenes.  Early  versions  of 
ACT-R  only  mimicked  intelligence  because  the  knowledge  is  instantiated  by  the  programmer 
and  is  not  learned  through  an  interaction  of  the  software  with  the  real  world.  More  recent 
approaches,  such  as  Symbolic  and  Sub-symbolic  Robotic  Intelligence  Control  System  (SS- 
RICS),  are  being  developed  to  control  small  robots  conducting  simple  tasks  such  as  walking 
through  a  door  and  finding  an  object  hidden  in  the  room  (Kelley,  2006;  Avery  et  al.,  2006).  SS- 
RICS  uses  production  systems  and  semantic  networks  for  knowledge  representations  and  various 


8 


learning  algorithms  that  allow  the  robot  to  generalize  its  training  trials  to  new  situations.  There 
are  disadvantages  to  the  cognitive  approach  as  well;  for  many  tasking  environments,  more 
efficient  algorithms  are  available  without  the  complexity  inherent  in  cognitive  approaches.  The 
advantage  of  cognitive  approaches  is  that  such  approaches  use  machine  learning,  neural  nets,  and 
NLP  algorithms  that  are  compatible  with  human  knowledge  structures  (Jurafsky  and  Martin, 
2009;  Liu  and  Singh,  2004;  Vinokurov  et  al.,  201 1).  Human-like  knowledge  structures 
combined  with  NLP  capabilities  are  essential  if  agents  are  to  develop  a  peer-to-peer  relationship 
in  open-ended  environments  (Ball  et  al.,  2010).  ACT-R  and  NLP  software  are  the  basis  of  a 
synthetic  teammate  research  project  to  emulate  a  unmanned  aerial  vehicle  (UAV)  pilot  with  an 
intelligent  agent  that  is  capable  of  generating  natural  dialogue  with  human  team  mates  as  well 
conducting  UAV  flight  functions  (Ball  et  al.,  2010).  The  agent  is  not  autonomous  in  that  mission 
planning  and  mission  objectives  are  supplied  by  human  operators.  The  agent  is  being  designed 
not  only  to  interact  with  humans  but  to  be  a  part  of  a  UAV  crew  during  realistic  simulations — 
suggesting  the  possibility  of  humans  and  agents  forming  peer-to-peer  relationships  in  the  future. 

3.2  Teaming  Agents 

In  recent  years,  researchers  have  designed  agents  that  support  team  behaviors.  Korsah  et  al., 
(2012),  for  example,  presented  a  multi-robot  control  system,  called  xBots,  that  could  support 
optimal  task-allocation  planning  and  flexible  execution  of  the  plans  as  well  as  allow  graceful 
degradations  of  the  plans  should  failures  or  unexpected  events  happen.  One  key  difference 
between  the  xBots  system  and  other  market -based  approaches  is  that  it  optimizes  the  plans  based 
on  temporal  constraints  in  the  tasking  environments.  In  order  to  facilitate  natural  interaction 
between  humans  and  robots/agents,  dialog  management  systems  such  as  TeamTalk  (Rudnicky 
et  al.,  2010)  have  also  been  developed.  The  effectiveness  of  TeamTalk  has  been  evaluated  in  a 
simulated  environment  that  involved  multiple  robots  for  search  and  rescue  tasks  (Marge  et  al, 
2009).  Other  experimenters  found  that  the  type  of  communication  among  agents  and  human 
teams  was  important  (Harbers  et  al.,  2012).  For  example,  Harbers  et  al.  found  that 
communication  with  agent  team  members  concerning  world  knowledge  of  the  tasking 
environment  was  not  as  effective  as  communicating  the  human’s  intent. 

Other  H-A  teaming  research  and  development  efforts,  while  not  directly  applied  to  multi-robot 
control  domains,  are  potentially  useful  for  developing  intelligent  agents  that  can  function  as 
teammates  to  the  humans  in  multi-robot  tasking  contexts.  For  example,  Fan  et  al.,  (2010) 
presented  research  on  agents  that  can  support  the  development  of  shared  mental  models  (see  also 
Nikolaidis  and  Shah,  2012).  These  researchers  discussed  the  use  of  the  Collaborative  Agents  for 
Simulating  Teamwork  (CAST)  model  based  on  the  RPD  framework  (R-CAST),  which  enables 
agents  and  humans  to  anticipate  potential  information  needs  and  proactively  exchange 
information  between  team  members.  The  R-CAST  agent  architecture  is  based  on  the  theory  of 
shared  mental  models  in  human  teams,  proactive  information  distribution,  and  RPD  (Klein, 
2008).  In  essence,  the  objective  of  the  R-CAST  agent  is  to  facilitate  the  “collaborative-RPD” 
process  between  the  human  operator  and  the  agent  by  reasoning  and  representing  the  “contextual 
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needs  of  collaboration”  in  the  following  areas:  decision  process  (establishing  shared  mental 
models  between  the  human  and  the  agent),  recognition  aspects  (cues,  goals,  expectancy,  and 
courses  of  action),  and  the  inference  mechanism.  Fan  et  al.,  (2010)  conducted  a  human-in-the- 
loop  simulation  experiment  and  demonstrated  that  the  H-A  teams  outperformed  the  human- 
human  (H-H)  teams  in  complex  military  command-and-control  decision-making  tasks,  regardless 
of  the  time  pressure  imposed  on  the  teams.  However,  the  H-A  team  performance  was  more 
sensitive  to  the  changes  of  tasking  complexities  than  the  H-H  teams  (i.e.,  degraded  faster  when 
tasks  were  more  complex)  than  the  H-H  teams.  The  results  of  the  Fan  et  al.,  (2010)  suggest  that 
using  collaboration  models  such  as  R-CAST  to  form  H-A  teams  may  lead  to  better  performance, 
although  the  overall  H-A  team  performance  may  be  moderated  by  tasking  complexities. 

3.3  Hierarchical  Agents 

In  complex  tasking  environments,  individual  agents  usually  lack  flexibility  and  thus  may  be  able 
to  perform  only  very  circumscribed  problem  sets.  For  more  complex  problems,  hierarchical 
agent  systems  are  being  designed  with  agents  that  have  specialized  intelligence  embedded  within 
multi-layer  architectures.  The  individual  agents  have  specific  tasks  and  a  means  of 
communicating  with  other  agents.  The  ability  to  divide  task  complexity  between  senior  (more 
capable)  agents  and  specialized  (less  capable)  ones  allows  hierarchies  to  adjust  to  greater 
complexity  and  better  adjust  to  change.  The  disadvantages  are  that  as  hierarchies  become  more 
complex,  the  algorithms  to  control  them  also  become  more  cumbersome,  and  even  then,  agent 
hierarchies  are  still  challenged  by  truly  novel  situations.  Furthermore,  the  more  levels  involved 
and  the  more  entities  that  need  to  be  controlled,  the  more  likely  it  is  that  communications  among 
agents  will  become  a  serious  problem. 

Agent  technologies  with  military  import  have  been  demonstrated  for  a  number  of  realistic 
applications.  For  example,  agents  have  been  successfully  used  to  control  multiple  UAVs  with 
received  signal  indicator  sensors  to  locate  targets  cooperatively  during  high  fidelity  simulations 
(Scerri  et  al.,  2008).  Researchers  for  the  Defence  Research  &  Development  Canada 
demonstrated  the  utility  of  agent  hierarchy  technology  (Adaptive  Intelligent  Agent  [AIA])  for 
UAV  control  in  a  more  complex  simulation  environment.  Hou  and  his  colleagues  (Hou  et  al., 
2011)  used  senior  agents  directing  working  agents  which,  in  turn,  directed  junior  agents.  Their 
simulation  demonstrated  agents  in  concert  with  the  operator,  planning  multiple  UAV  missions, 
navigating  UAVs  to  the  target  area,  and  upon  arrival  directing  sensors  to  locate  the  targets. 
Similar  agent  technology  (e.g.,  the  Cognitive  and  Cooperative  Assistant  System  [CCAS]),  has 
been  used  for  cooperative  control  of  multiple  UAVs  using  live  UAV  demonstrators  (Strenzke, 
2011). 

3.4  Flexible  Automation 

Flexible  automation  refers  to  systems  that  are  neither  fully  automated  nor  fully  manual  but  that 
invoke  various  levels  of  automation  depending  on  the  operator’s  state,  critical  events  in  the 
environment,  or  algorithms  related  to  specialized  problem  sets.  The  approaches  differ  in  their 
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workload  requirements  (amount  of  cognitive  resources  available  for  tasks)  and  allocation  of 
decision  authority  (Goodrich,  2010).  Flexible  automaton  is  a  useful  set  of  concepts  for  eliciting 
agents  while  minimizing  workload  depending  on  the  requirements  of  the  specific  tasks  in  which 
agents  are  being  used.  The  difference  between  the  types  of  flexible  automation  involves  the  role 
of  the  human  in  decision  making:  (1)  adaptive  systems  assign  tasks  under  specified  conditions 
decided  on  before  the  mission,  (2)  adjustable  (also  known  as  adaptable)  automation  requires 
humans  to  decide  when  to  invoke  automation  during  the  mission,  whereas  (3)  mixed  initiative 
systems  entail  joint  decision  making  (e.g.,  the  planning  process)  throughout  the  mission 
(Parasuraman,  2007;  Miller  and  Parasuraman,  2007;  Tecuci  et  al.,  2007). 

3.4.1  Adaptive  Automation 

Adaptive  systems  are  being  developed  to  maintain  a  balance  between  reducing  workload  for 
manual  control  and  reducing  SA  loss  for  automated  tasks  (Kaber  et  al.,  2006).  Adaptive 
automation  is  most  effective  for  multitasking  situations  wherein  a  fully  automated  task  may  be 
neglected  if  the  operator  becomes  complacent  or  allocates  attentional  resources  to  other  tasks.  A 
trigger  based  on  environmental  or  internal  state  indicators  is  used  to  allocate  tasks  to  either 
automation  or  manual  control.  During  low  workload,  the  task  is  allocated  to  the  human  and 
during  high  workload  conditions  the  task  is  automated.  The  advantage  compared  to  full 
automation  is  that  it  keeps  the  human  engaged  but  not  overwhelmed.  Physiological  measures 
(e.g.,  electroencephalography  [EEG],  functional  magnetic  resonance  imaging,  heart  rate)  have 
been  used  successfully  in  a  number  of  laboratory  settings  as  triggers  to  different  workload  states 
as  well  as  to  improve  tracking  performance  (using  switching  rates  of  as  low  as  15s  between 
automated  and  human  control).  It  has  also  been  used  for  adapting  operator  performance  when 
controlling  UAVs  (Prinzel,  2000;  Wilson  and  Russell,  2003).  Operator  error  rates  and  task 
difficulty  have  been  used  as  successful  triggering  mechanism  to  improve  secondary  tasks 
performance,  to  reduce  mental  workload  and  improve  SA  for  supervisory  control  of  UAV  and 
unmanned  ground  vehicle  (UGV)  combined  missions  (Parasuraman  et  al.,  2007).  Because  modem 
neurophysiological  devices  are  becoming  more  cost  effective  and  less  invasive,  future  systems  will 
most  likely  take  advantage  of  this  technology  to  monitor  workload  states  (Barnes  et  al.,  2006). 

There  are  a  number  of  technical  and  philosophical  issues  with  adaptive  systems  (Miller  and 
Parasuraman,  2007).  If  the  trigger  is  less  than  perfect,  then  sudden  changes  in  task  state  may  be 
annoying  or  even  dangerous.  Furthermore,  adaptive  systems,  by  their  very  nature,  usurp 
delegation  authority  from  the  human.  On  the  other  hand,  it  reduces  the  task  management  load; 
the  operator  does  not  have  the  additional  task  of  deciding  when  or  what  tasks  to  automate 
(Goodrich,  2010).  More  importantly,  when  there  are  safety  issues  and  time  constraints  such  as  a 
missile  in-flight,  the  operator  may  not  have  the  luxury  of  giving  permission  before  the  automated 
systems  are  activated  (Bames  and  Grossman,  1985).  Parasuraman  et  al.,  (2009)  and  Steinhauser 
et  al.,  (2009)  provided  several  design  guidelines  for  implementing  adaptive  automation  based  on 
their  own  experimental  results  as  well  as  reviews  of  empirical  research  on  adaptive  automation 
and  aiding  (Chen  et  al.,  201 1). 
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3.4.2  Adjustable  Automation 

In  adjustable  (or  adaptable)  automated  systems,  in  contrast  to  adaptive  automation,  the  adaptivity 
of  the  joint  human-system  task  is  initiated  by  the  human  (as  opposed  to  the  automated  systems  in 
the  case  of  adaptive  automation  (Goodrich,  2010;  Langan-Fox  et  al.,  2009;  Miller  and 
Parasuraman,  2007).  The  precise  level  of  control  is  a  matter  of  degree — we  use  a  variation  of  the 
levels  of  autonomy  (LOA)  in  Parasuraman  et  al.,  (2000),  designating  LOA-6  as  the  cut-off  point 
for  adjustable  automation:  the  human  operator  must  be  given  time  and  opportunity  to  veto  an 
automation  decision  or  make  a  decision  on  whether  to  automate.  Thus,  the  machine  can  make 
decisions,  but  only  the  human  can  give  permission  to  execute  them.  Miller  and  Parasuraman 
(2007)  have  developed  an  adjustable  system  they  refer  to  as  adaptable  wherein  specified 
algorithms  known  as  plays  can  be  elicited  by  the  operator  to  improve  performance  during  high 
workload  mission  segments.  The  plays  can  be  instantiated  either  because  they  represent 
algorithms  that  can  be  handled  better  by  machine  intelligence  or  to  reduce  workload.  The 
delegation  model  incorporated  in  their  Playbook*  regimen  is  specific  to  a  task  environment  with 
a  hierarchy  of  plays  designed  to  obtain  goals  important  to  that  environment.  It  has  been  used 
successfully  for  gaming  situations  such  as  RoboFlag,  a  capture  the  flag  game  using  robots 
(Miller  and  Parasuraman,  2007).  In  a  recent  experiment,  Miller  et  al.,  (201 1)  examined  the 
effects  of  non-optimal  Playbook  solutions  on  operator  performance.  More  germane  to  our 
discussion,  the  Playbook  paradigm  has  also  been  used  successfully  to  simulate  UAV  flights  and 
has  been  used  in  simulation  and  field  testing  of  UAV  automation  for  the  U.S.  Army  (Fem  and 
Shively,  2009). 

3.4.3  Mixed-Initiative  Automation 

Mixed  initiative  automation  involves  collaborative  decision  making  between  a  human  and  an 
intelligent  system  much  like  the  relationship  between  a  human  and  a  subordinate  but  autonomous 
assistant  (Cummings  et  al.,  2012;  Tecuci  et  al.,  2007).  For  example,  the  Mixed-initiative 
Activity  Plan  GENerator  (MAPGEN)  was  developed  by  National  Aeronautics  and  Space 
Administration  (NASA)  to  aid  planners  for  the  Mars  space  mission  (Bresina  and  Morris,  2007). 
MAPGEN  used  constraint  reasoning  to  generate  mission  plans  for  the  Mars  mission  based  on 
priorities  input  by  human  planning  experts  considering  science  objectives  as  well  as  temporal 
and  physical  mission  constraints.  Initially,  the  constraints  were  considered  “hard”  parameters 
and  the  human  could  not  override  them  even  temporarily  (although  the  final  plan  had  to  be 
approved  by  a  technical  committee).  In  subsequent  software  builds,  MAPGEN  constraints  were 
relaxed,  allowing  human  planners  more  flexibility  in  creating  what-if  scenarios,  resulting  in  a 
more  synergistic  relationship  between  humans  and  MAPGEN.  Another  example  examined  the 
utility  of  a  mixed-initiative  system  in  the  context  of  a  simulated  search-and-rescue  task  (Hardin 
and  Goodrich,  2009).  The  study  involved  simulating  200  robots  searching  a  large  wilderness 
area  for  five  lost  hikers.  For  “adaptive”  conditions,  the  decision  determining  the  robots’  search 
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pattern  is  triggered  by  discovery  of  cues  relating  to  the  hiker’s  location.  In  “adjustable” 
conditions,  the  human  supervisor  decides  on  the  level  of  search  responsibility  for  the  robots.  For 
the  “mixed-initiative”  condition,  either  the  human  or  the  robot  can  decide  search  responsibility 
when  a  cue  is  found  depending  on  whom  discovered  the  cue.  In  general,  the  mixed- initiative 
condition  resulted  in  the  best  performance  especially  for  the  primary  target  detection  task. 

Tecuci  et  al.,  (2007)  point  out  that  designing  a  mixed-initiative  system  requires  consideration  of 
seven  issues:  task  parameters,  control,  shared  awareness,  communications,  personalization, 
software  architecture,  and  evaluation.  This  suggests  that  the  system  designers  should  not  only 
consider  the  relative  technical  merits  of  the  software  agents  and  the  human  operators’ 
capabilities  but  also  the  ability  of  both  the  agents  and  the  humans  to  communicate  their  intents  as 
well  as  the  mixed- initiative  system’s  ability  to  respond  to  developments  in  the  tasking 
environment. 

3.4.4  Combining  Architectures 

In  reading  the  different  descriptions  of  flexible  automation,  it  is  obvious  that  the  different 
varieties  are  often  blurred.  In  practice,  adaptive  systems  can  also  be  adjustable  or  even  mixed. 
The  rotorcraft’s  Pilot  Associate  program  allowed  a  pilot  to  set  parameters  before  the  mission  that 
determined  under  what  conditions  an  adaptive  process  would  be  invoked  (Parasuraman  et  al., 
2007).  Barnes  and  Grossman  (1985)  gave  an  example  of  a  situation  that  was  time  sensitive.  If  a 
missile  was  noted  by  a  pilot  within  a  short  time  window  (a  few  seconds),  then  the  pilot  could 
take  evasive  action;  however,  when  the  time  window  expired,  the  automated  systems  performed 
the  maneuver  without  waiting  for  the  pilot’s  permission.  This  was  because  the  maneuver  was 
useless  when  the  deadline  expired;  the  missile  would  inevitably  destroy  the  plane  if  the  pilot  had 
failed  to  act.  The  point  is  that  the  different  types  of  automation  under  investigation  can  be 
combined  into  complex  architectures  to  deal  with  the  real-world  complexity.  More  complicated 
architecture  such  as  the  one  suggested  by  Hou  et  al.,  (2011;  discussed  in  section  3.3)  that 
combine  different  types  of  flexible  automation  with  various  levels  of  intelligent  agents  are  more 
likely  to  be  used  by  designers  of  real-world  systems.  Real-world  systems  must  conduct  many 
types  of  missions  under  volatile  conditions  and  be  able  to  react  to  unforeseen  circumstances 
rapidly.  The  latter  constraint  is  an  argument  for  the  importance  of  H-A  teaming  within  a  mixed- 
initiative  paradigm.  Humans  having  final  decision  authority,  however,  does  not  imply  that 
humans  cannot  delegate  the  authority  to  agents  under  specified  conditions  (Parasuraman  et  al., 
2007). 

3.5  RoboLeader 

Past  research  indicates  that  autonomous  cooperation  between  robots  can  improve  the 
performance  of  the  human  operators  (Lewis  et  al.,  2010)  as  well  as  enhance  the  overall  human- 
robot  team  performance  (Schurr,  2007).  Researchers  have  proposed  a  number  of  solutions  to  the 
potential  issues  of  a  robotic  battlefield,  such  as  setting  up  a  robot  call  center  in  which  robots 
query  the  human  operator  only  when  there  is  a  problem  (Xu  et  at.,  2012).  The  operator  makes 
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the  necessary  adjustments  but  is  not  required  to  monitor  robots  continuously.  A  potential  issue 
with  this  solution  is  that  it  requires  the  robots  to  be  capable  of  self-diagnosing  their  own 
problems  with  high  accuracy;  additionally,  the  number  of  operator-robot  interactions  is  expected 
to  increase  exponentially  during  the  heat  of  combat,  making  the  call  center  ineffective  during  the 
most  critical  time  periods.  Goodrich  (2010)  concluded  that  as  the  number  of  robots  increases, 
there  will  be  a  maximum  number  that  the  human  can  monitor  or  control  directly  depending  on 
the  complexity  of  the  environment  or  the  state  of  the  operator.  Wang  and  his  colleagues  (Wang 
et  al.,  2009)  reviewed  a  number  of  studies  on  supervisory  control  of  multiple  ground  robots  for 
target  detection  tasks  and  concluded  that  in  order  to  be  effective,  “the  Fan-out  plateau  lies 
somewhere  between  4  and  9+  robots  depending  on  the  level  of  robot  autonomy  and 
environmental  demands”  (p.  143),  limits  very  close  to  the  classical  (7  ±  2)  findings  for  span  of 
apprehension  (Miller,  1956).  Obviously,  the  human  operator’s  attention  issue  is  even  more 
problematical  when  the  operator  must  multitask  as  well  as  control  the  robot  (Barnes  and  Evans, 
2010;  Chen  and  Barnes,  2012a;  Chen  et  al.,  2011;  Chen  and  Terrence,  2008).  Nevertheless, 
simply  increasing  the  robots’  level  of  autonomy  does  not  resolve  the  issue.  Since  human 
operators’  decision  making  may  be  influenced  by  “implicit  goals”  that  the  robots  are  not  aware 
of  (i.e.,  are  not  programmed  into  the  behaviors  of  the  robots;  Linegang  et  al.,  2006),  human 
operators’  involvement  in  mixed-initiative  teams  will  still  be  required  for  the  foreseeable  future, 
especially  in  situations  involving  critical  decision  making.  In  other  words,  the  human  must 
always  be  in  ultimate  control  for  safety  as  well  as  flexibility  reasons. 

In  order  to  evaluate  some  of  the  issues  associated  with  intelligent  agents  for  multi-robot  control, 
the  U.S.  Army  developed  RoboLeader,  an  agent  that  can  help  the  human  operator  coordinate  a 
team  of  robots,  as  a  research  paradigm  (Chen  and  Barnes,  2012a).  Instead  of  directly  managing 
each  individual  robot,  the  human  operator  only  deals  with  one  single  entity — RoboLeader.  The 
operator  can,  therefore,  better  focus  on  other  tasks  requiring  his/her  attention.  More  specifically, 
RoboLeader  can  collect  information  from  subordinate  robots  with  limited  autonomy  (e.g., 
collision  avoidance  and  self-guidance  to  reach  target  locations),  making  tactical  decisions  and 
coordinating  the  robots  by  issuing  commands,  waypoints,  or  motion  trajectories  (Snyder  et  al., 
2010).  In  typical  mission  situations,  RoboLeader  would  recommend  route  revisions  when 
encountering  environmental  events  that  require  robots  to  be  rerouted.  The  human  operators,  in 
turn,  can  accept  the  plan  revisions  or  modify  them  as  appropriate.  RoboLeader,  therefore,  is  a 
mixed-initiative  system  that  consists  of  both  human  and  machine  decision-making  components. 
It  also  possesses  characteristics  of  hierarchical  systems  as  RoboLeader  serves  as  the  interface 
between  the  human  supervisor  and  the  less  capable  (in  terms  of  decision  authority)  robots. 

A  series  of  human-in-the-loop  simulation  experiments  have  been  conducted  to  investigate  the 
control  structure  and  interface  requirements  between  the  human  supervisor  and  RoboLeader, 
with  number  of  robots,  task-load,  target  mobility,  agent  error  type,  and  agent  reliability  level 
being  manipulated  systematically  (Chen  and  Barnes,  2012a;  Chen  and  Bames,  2012b).  The  first 
experiment  established  the  feasibility  of  using  RoboLeader  to  control  up  to  eight  robots  during  a 
reconnaissance  mission  (Chen  and  Bames,  2012a).  The  results  indicated  equivalent  target 
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detection  between  RoboLeader  and  baseline  conditions  while  reducing  mission  times  by  12%  for 
the  RoboLeader  condition.  The  second  experiment  focused  on  RoboLeader’s  reliability  level 
(60%  versus  90%)  and  type  of  possible  errors  (false  alarm-prone  [FAP]  versus  miss-prone  [MP]; 
Chen  and  Barnes,  2012a).  Results  showed  that  the  type  of  RoboLeader  unreliability  (FAP  versus 
MP)  affected  operator’s  performance  of  visual  scanning  tasks  (target  detection,  route  editing,  and 
SA).  Overall,  MP  RoboLeader  was  more  detrimental  than  FAP  RoboLeader,  although, 
paradoxically,  participants  showed  better  SA  of  the  tasking  environment  due  to  constant 
scanning  of  the  environment.  The  final  experiment  demonstrated  the  efficacy  of  RoboLeader  in 
aiding  the  operator  to  conduct  more  dynamic  missions,  which  required  four  robots  to  entrap  a 
moving  target  (Chen  and  Bames,  2012b).  Results  showed  that  the  participants’  target 
encapsulation  task  benefited  from  RoboLeader  during  all  LOA  conditions  compared  with  manual 
performance,  and  that  the  intermediate  LOA  is  sufficiently  beneficial.  Participants  successfully 
encapsulated  the  moving  targets  only  63%  of  the  time  in  the  manual  condition  but  89%  of  the 
time  when  they  were  assisted  by  RoboLeader.  Additionally,  participants  reported  significantly 
higher  workload  when  they  were  in  the  manual  condition  than  when  they  were  assisted  by 
RoboLeader.  Across  all  three  experiments,  participants  with  higher  spatial  ability  consistently 
outperformed  those  with  lower  spatial  ability  in  tasks  that  required  the  most  visual  scanning, 
regardless  of  the  availability  of  RoboLeader  and  other  experimental  manipulations.  Participants 
with  higher  attention  control  ability  and  those  who  played  video  games  frequently  were  able  to 
multitask  better  than  their  counterparts.  Frequent  gamers  also  repeatedly  exhibited  better  SA  of 
the  mission  environments  than  did  infrequent  gamers.  Further  discussion  on  these  individual 
differences  issues  will  be  presented  in  section  6. 

In  general,  the  RoboLeader  research  suggests  a  synergy  between  humans  and  intelligent  agents 
but  not  an  equivalency.  The  agents  were  ideal  for  circumscribed  solutions  and  as  means  to 
reduce  the  operator’s  burden  of  multitasking.  However,  it  is  essential  to  maintain  the  human’s 
ultimate  decision  authority  without  affecting  their  awareness  of  the  unfolding  combat  situation. 
The  following  section  will  review  some  of  the  key  human  performance  issues  in  H-A  teaming  as 
well  as  potential  user  interface  design  solutions  to  mitigate  those  issues. 


4.  Human  Performance  Issues  in  H-A  Teaming:  Trust  in  Automation 


In  mixed- initiative  operations,  the  human  operator’s  trust  in  the  automated  systems  is  a  critical 
element  of  the  H-A  team  performance.  There  have  been  numerous  definitions  of  trust  in  the 
literature  of  organization  psychology,  interpersonal  relationship,  human-machine  interactions, 
just  to  name  a  few  (Lee  and  See,  2004;  Madhavan  and  Wiegmann,  2007;  Mayer  et  al.,  1995; 
Rempel  et  al.,  1985).  In  the  context  of  H-A  interaction,  Lee  and  See’s  definition  of  trust  seems 
most  relevant:  “the  attitude  that  an  agent  will  help  achieve  an  individual’s  goals  in  a  situation 
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characterized  by  uncertainty  and  vulnerability”  (p.  54).  According  to  Mayer  et  al.,  (1995)  trust 
model  (in  the  context  of  organization  psychology),  there  are  four  major  antecedents  that 
contribute  to  trust  development:  the  trustor’s  propensity  to  trust  and  the  trustee’s  ability, 
benevolence,  and  integrity.  Lee  and  See  (2004),  on  the  other  hand,  use  the  tenns  “performance, 
purpose,  and  process”  to  describe  the  aspects  of  “ability,  benevolence,  and  integrity”  for  the 
discussion  of  trust  developments  in  the  context  of  human- automation  interaction.  It  should  be 
pointed  out  that  over  trust  is  as  deleterious  as  under  trust.  Our  assumption  is  that  appropriate 
trust  implies  that  humans  understand  an  agent’s  limits  as  well  as  its  benefits.  The  following 
sections  review  the  system  and  environment -related  factors  as  well  as  human-related  factors  in 
H-A  trust.  Potential  system-design  principles  derived  from  research  findings  are  also  reviewed. 

4.1  System  and  Environment-Related  Factors 

Lee  and  See’s  (2004)  identification  of  antecedents  for  trust  development  in  the  context  of 
human-automation  interaction  (i.e.,  purpose,  process,  and  performance  (3P’s)  provides  a 
framework  for  discussion  of  system  and  environment-related  factors  that  contribute  to  trust 
development.  The  “purpose”  factor  deals  with  the  degree  to  which  the  automation  is  being  used 
according  to  the  designer’s  intent.  The  “process”  factor  deals  with  the  question  whether  the 
algorithm  of  the  automated  system  is  appropriate  for  a  given  tasking  situation.  The 
“performance”  factor  deals  with  system  reliability,  predictability,  and  capability.  A  number  of 
studies  have  examined  the  differences  in  people’s  perceived  reliability  of  human  versus 
automated  aids  (Dzindolet  et  al.,  2001;  Madhavan  and  Wiegmann,  2007).  Although  several 
factors  may  affect  the  perception  (e.g.,  the  context  of  the  decision-making  tasks,  the  operators’ 
self-confidence  in  perfonning  the  tasks,  the  operators’  monitoring  strategies  and  expectations  of 
performance  of  the  systems,  etc.),  it  was  found  that  people  tend  to  perceive  the  automation  as 
being  more  capable  and  reliable  than  the  human  aids  (when  in  fact  the  same  info  was  provided  to 
them).  However,  the  caveat  is  that  people  are  more  sensitive  to  the  automation’s  errors 
compared  with  another  human’s  errors;  this  relative  sensitivity  leads  to  a  more  rapid  drop  in  trust 
in  the  automated  aids  once  errors  are  detected.  Jian  et  al.,  (2000)  also  found  that  people  are  more 
willing  to  use  stronger  terms  to  describe  their  distrust  in  a  machine  versus  a  human. 

Finally,  Lyons  and  Stokes  (2012)  investigated  the  effects  of  risk  on  human  operators’  reliance  on 
decision  aids  coming  from  another  human  verses  an  automated  system.  They  found  that,  as  the 
risk  became  greater,  the  human  operators  tended  to  rely  more  on  the  automation  rather  than  the 
human  aid.  In  a  previous  study  conducted  by  the  same  group  of  researchers  dealing  with  similar 
tasking  environments  (computer-based  convoy  scenarios  involving  an  automated  decision  aid),  it 
was  reported  that  suspicion  concerning  the  information  technology  (e.g.,  abnormal 
activities/behaviors,  integrity  of  the  data,  validity  of  the  output,  access  to  the  data,  etc.)  had 
different  effects  on  human  operators’  self-assessed  decision  confidence  than  did  the  operators’ 
trust  and  distrust  in  the  automation  (Lyons  et  al.,  2011). 
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The  issues  related  to  trust  in  autonomous  systems  span  multiple  levels  and  encompass  the  social 
and  ethical  aspects.  It  is  an  ongoing  debate  in  the  HRI  community  regarding  the  locus  of 
responsibility  (human  versus  robot)  when  a  robot  commits  a  harmful  act  under  a  human’s 
command,  Kahn  et  al.,  (2012)  showed  that  a  robot  may  be  perceived  as  unethical  when  it  causes 
harms.  Safeguard  measures  have  been  proposed  to  prevent  robots  from  committing  unethical  or 
dangerous  acts  (Arkin  and  Ulam,  2012).  Finally,  as  intelligent  systems  become  increasing 
sophisticated  and  are  capable  to  learn/evolve  either  based  on  their  learning  algorithms  or  access 
to  information  from  other  networks  (e.g.,  cloud-based  systems),  it  is  imperative  to  examine  the 
implications  of  these  capabilities  on  operator  trust  in  the  systems.  Since  predictability  is  a 
critical  aspect  of  trust  development  and  maintenance,  any  learning  behaviors  may  prevent  the 
operators  from  properly  calibrating  appropriate  trust.  Section  4.3  will  further  discuss  some  of  the 
system  design  strategies  to  mitigate  these  issues. 

4.2  H-R  Factors 

When  humans  perceive  automated  systems  as  team  members,  their  attitudes  of  misuse  or  disuse 
should  develop  as  operators  become  familiar  with  the  system,  implying  that  the  ‘trust’  decision  is 
dependent  on  the  attitude  of  humans  towards  automated  systems.  A  distinction  important  to 
disuse  or  misuse  of  automation  is  the  difference  between  evaluation  errors  and  intent  errors.  For 
evaluation  errors,  the  operator  misperceives  the  optimal  solution  and  commits  a  calibration  error. 
For  intent  errors,  the  operator  is  aware  of  the  aids  superiority  but  still  chooses  to  “disuse” 
automation  in  order  to  maintain  control  over  the  decision  environment.  For  example,  intent 
errors  occurred  when  participants  were  aware  of  the  superiority  of  an  automated  target  detection 
aid  but  84%  of  them  still  tended  to  disuse  the  aid,  leading  to  suboptimal  targeting  decisions 
(Beck  et  al.,  2007).  This  “disuse”  error  can  be  partially  contributed  to  the  operator’s  self- 
confidence  in  manually  performing  the  task.  Research  showed  that  self-confidence  is  a  critical 
factor  in  moderating  the  effect  of  trust  (in  automation)  on  reliance  (on  the  automatic  system) 
(Chen  and  Terrence,  2009;  deVries  et  al.,  2003;  Lee  and  Moray,  1994).  Lee  and  Moray  found 
that  when  self-confidence  exceeded  trust,  operators  tended  to  use  manual  control.  When  trust 
exceeded  self-confidence,  automation  was  used  more.  Chen  and  Terrence  showed  that  there  is  a 
strong  interaction  between  the  type  of  automation  unreliability  and  participants’  self-assessed 
attentional  control,  which  will  be  discussed  in  the  Individual  Differences  section  (6.1). 

Researchers  suggest  that  the  psychological  context  of  the  decision  determines  the  tendency  of  the 
operator  to  disuse  (under  rely)  or  misuse  (over  rely  on)  automated  systems  (Parasuraman  and 
Riley,  1997).  In  a  series  of  experiments,  Dzindolet  et  al.,  (2001)  showed  that  by  a  simple  change 
in  decision  order,  “disuse”  of  an  automated  target  recognition  device  changed  to  “misuse”.  If 
participants  made  a  decision  before  being  informed  of  the  automated  solution,  they  tended  to  rely 
on  their  own  decisions  even  when  they  were  suboptimal;  whereas,  in  a  related  experiment  using 
a  similar  paradigm,  participants  tended  to  over  rely  on  the  device  whenever  the  automated 
solution  was  presented  “at  the  same  time”  as  the  target  scene.  One  explanation  is  that 
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participants  were  attempting  to  reduce  their  cognitive  load  to  maintain  reasonable  timeliness. 
However,  such  a  strategy  was  not  useful  in  the  first  experiment  (automated  solutions  being 
shown  after  the  participants  made  their  decisions)  because  considering  an  alternative  would 
increase  workload  by  requiring  operators  to  reconsider  their  original  decisions.  The  workload 
hypothesis  is  supported  by  complacency  research  that  indicates  operators  misuse  automation  in  a 
multitasking  environment  but  not  in  single  task  environment  (Parasuraman  et  al.,  1993).  It  has 
also  been  reported  that  both  the  cost  of  automation  errors  and  the  cost  of  verification  affect 
humans’  reliance  on  decision  aids,  and  the  effects  are  also  moderated  by  age  (Ezer  et  al.,  2008). 
Generally,  reliance  is  reduced  as  the  cost  of  error  increases  and  it  is  increased  as  the  cost  of 
verification  decreases.  However,  younger  adults  are  more  flexible  than  older  adults  in  changing 
their  reliance  behavior  based  on  the  changing  cost  structure.  Since  the  older  participants  in  the 
Ezer  et  al.,  (2008)  study  were  over  65  years  old,  it  remains  to  be  seen  whether  a  similar  age 
effect  can  be  observed  for  younger  individuals. 

Human  operators’  attitudes  toward  and  trust  in  automated  systems  can  also  be  influenced  by  their 
personalities  (introversion  versus  extroversion;  Merritt  and  Ilgen,  2008)  and  affective  factors 
(e.g.,  moods  and  emotions;  Merritt,  2011).  Generally,  Merritt  and  Ilgen  found  that  extroverts 
tend  to  demonstrate  a  stronger  propensity  to  trust  the  automation  than  do  introverts  but  their  trust 
also  decline  more  readily  if  the  system  fails  to  behave  as  they  expect.  Merritt  (201 1)  found  that 
moods  (e.g.,  happiness)  significantly  impacted  the  human  operators’  trust  in  an  X-ray  screening 
decision  aid  system,  which  was  a  better  predictor  of  the  operators’  actual  reliance  on  the  system 
than  was  the  operators’  perceived  accuracy  of  the  system. 

Recently,  researchers  have  suggested  that  the  trust  in  automation  research  addressed  above  is  a 
foundation  for  modeling  trust  in  human-robot  interactions  (Desai  et  al.,  2009).  Desai  et  al. 
suggest  that  the  model  account  for  the  differences  between  current  systems  and  those  used  in 
prior  research;  for  example,  the  newer  systems’  levels  of  reliability  and  the  more  dynamic  nature 
of  the  systems  in  terms  of  adjustable  autonomy  and  their  ability  to  carry  out  multiple  tasks. 

There  have  also  been  efforts  on  developing  training  tools  for  reducing  operator  errors  associated 
with  disuse  or  misuse.  For  example,  Beck  et  al.,  (2007)  suggest  that  a  combination  of 
performance  feedback  and  scenario  training  can  reduce  both  appraisal  and  intent  errors. 

4.3  Implications  for  System  Design — Trust  Calibration 

Lessons  learned  from  a  U.S.  Naval  Intelligent  Autonomy  program  indicated  that  human 
operators  sometimes  questioned  the  accuracy  and  effectiveness  of  the  output  produced  by 
intelligent  systems  such  as  those  generating  automated  plans  (Linegang  et  al.,  2006).  For 
example,  some  human  operators  indicated  that  they  had  difficulties  understanding  the  rationales 
for  some  plans  and  how  the  plans  were  generated  (Billman  et  al.,  2005,  as  cited  in  Linegang 
et  al.,  2006).  Additionally,  some  operators  reported  that  they  had  difficulties  at  times  when 
trying  to  specify  mission  parameters  (e.g.,  goals  and  constraints)  in  the  way  required  by  the 
automated  planning  system  (Billman  et  al.,  2005).  Furthermore,  real-time  development  on  the 
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battlefield  may  require  the  human  operator  to  change  the  plan  for  the  robot  team  and/or  for  the 
individual  robots.  Lee  and  See  (2004)  recommended  that  the  capabilities  and  limitations  of  the 
automated  systems  be  conveyed  to  the  operator,  when  feasible,  in  order  for  the  operator  to 
develop  appropriate  trust  and  reliance.  For  example,  McGuirl  and  Sarter  (2006)  found  that 
pilots’  trust  in  an  automated  decision  aid  system  was  better  calibrated  when  dynamic  system 
confidence  information  was  provided  to  them.  Similarly,  Seong  and  Bisantz  (2008)  reported  that 
participants  performed  a  target  identification  task  significantly  better  and  their  trust  better 
calibrated  if  provided  with  “meta- information”  that  informed  them  about  the  performance  of  the 
decision  aid  than  those  participants  without  the  information.  Bagheri  and  Jamieson  (2004) 
demonstrated  that  when  operators  were  aware  of  the  context-related  nature  of  automation 
reliability,  their  detection  rate  of  automation  failures  increased  significantly  without  affecting 
their  concurrent  tracking  and  system  management  tasks.  The  authors  attributed  this 
improvement  in  performance  to  a  more  effective  attentional  allocation  strategy.  Based  on  their 
findings  on  the  different  types  of  unreliable  automation,  Rovira  et  al.,  (2007)  suggested  that 
decrements  in  decision-making  performance  may  be  mitigated  if  operators  can  query  the 
automation,  inspect  raw  information  sources,  and  verify  or  negate  the  automated  advice.  This 
remedy  can  potentially  mitigate  the  “merging  trust”  phenomenon  described  in  Keller  and  Rice 
(2010)  where  operator’s  trust  in  higher-reliability  system  was  “pulled-down”  by  lower-reliability 
system  (i.e.,  system-wide  trust  calibration  versus,  component-specific  calibration).  Roth  et  al., 
(2004)  also  reported  that,  in  order  to  enhance  the  mixed-initiative  team  performance,  the 
automated  system  should  communicate  to  the  human  operator  about  its  rationale  for  the  system¬ 
generated  plans  while  allowing  the  operator  flexibility  in  comparing  different  alternatives  and 
testing  “what-if  ’  scenarios.  While  excessive  alerts  may  create  the  cry-wolf  effect,  Lees  and  Lee 
(2007)  did  find  that  “unnecessary  alarms”  (i.e.,  alerts  that  are  legitimate  but,  due  to  the 
peculiarities  of  the  situation,  do  not  require  operator  compliance)  foster  the  operators’  trust  in  the 
automated  system  and  actually  enhance  their  compliance  with  the  alerts  rather  than  reducing  it. 
Lees  and  Lee’s  data  suggest  that  the  three  dimensions  of  trust  (utility,  predictability,  and  intent) 
need  to  be  considered  beside  the  traditional  description  of  alarms  according  to  signal  detection 
theory  (e.g.,  false  alarm-prone  or  miss-prone). 

Seppelt  and  Lee  (2007)  designed  a  display,  based  on  the  ecological  interface  design  principles, 
which  graphically  portrayed  the  capabilities  and  limitations  of  a  vehicle’s  adaptive  cruise  control 
for  driving  in  various  traffic  and  weather  conditions.  They  found  that  drivers’  reliance  on  the 
automation  (i.e.,  cruise  control)  was  more  appropriate  when  the  display  was  present  than  when  it 
was  not.  In  another  study  conducted  by  Lee’s  group  on  supervisory  control  of  multiple  robots 
with  the  assistance  of  an  intelligent  agent,  it  was  reported  that  display  transparency  (overlays  that 
portray  robot  status,  target-related  info,  communications  aspects,  etc)  not  only  enhanced  the 
overall  H-A  team  performance  but  also  enabled  better  operator  calibration  of  appropriate  trust 
(reliance  and  compliance)  in  the  agent.  Lee  (2012)  proposed  that,  in  order  to  increase 
automation  transparency  to  the  operator,  system  designers  should  make  the  3P’s  as  well  as  the 
history  of  the  3P’s  visible  to  the  operator.  However,  the  presentation  should  be  in  a  simplified 
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form,  so  the  operator  is  not  overwhelmed  by  the  amount  of  information  to  process.  Neyedli 
et  al.,  (201 1)  showed  that  integrated  displays  that  portray  system  reliability  info  and  other  task- 
related  info  (compared  with  separate  displays  showing  the  same  info)  were  more  effective  in 
supporting  appropriate  automation-reliance  behaviors  in  their  simulated  target  identification 
tasks.  Cook  and  Smallman  (2008)  also  demonstrated  the  effectiveness  of  an  integrated  graphical 
display  that  promoted  more  comprehensive  consideration  of  task-related  info  (i.e.,  less 
confirmation  bias).  Lee  and  See  (2004)  recommend  that  the  automated  system  should  propose  a 
family  of  potential  solutions,  rather  than  just  the  optimal  one.  The  user  interface  should  also 
enable  flexible  adjustments  of  weighting  of  constraints  via  a  user-friendly  mechanism  (e.g., 
sliders).  Finally,  Lee  and  See  suggest  that  operators  should  be  trained  to  understand  the  system’s 
“expected  reliability,  the  mechanisms  governing  its  behavior,  and  its  intended  use”  (p.  74). 

Cummings  et  al,.  (2010)  evaluated  an  intelligent  path-planning  agent  that  assisted  the  human 
operators  with  a  simulated  maritime  navigation  task.  The  user  interface  of  the  planner  agent  was 
designed  based  on  the  objectives  of  promoting  automation  transparency  and  conveying 
environmental  uncertainties  while  providing  the  participants  with  flexibility  in  specifying 
constraints  for  the  solutions.  Cummings  et  al.,  (2010)  reported  that,  while  the  participants  found 
the  planner  agent  highly  usable  in  terms  of  capabilities  and  reliability,  they  only  rated  it  as 
somewhat  trustworthy.  However,  the  authors  suggest  that,  for  mixed-initiative  system  such  as 
their  planner  agent,  this  level  of  trust  is  actually  appropriate  (i.e.,  not  over-  or  under-trust),  given 
the  risks  involved  in  the  tasking  environments. 

Finally,  Madhavan  and  Wiegmann  (2007),  based  on  the  research  findings  of  Nass  and  Moon 
(2000)  on  humans’  social  responses  to  computers,  suggested  that  techniques  such  as 
anthropomorphizing  of  automation  may  help  individuals  better  calibrate  their  trust  in  the 
systems.  Specifically,  the  “humane”  responses/characteristics  exhibited  by  automation  may 
reduce  users’  biases  toward  machines  and  facilitate  their  application  of  reciprocal  behaviors. 
However,  Lee  and  See  (2004)  cautioned  that  any  anthropomorphizing  of  the  automation  should 
be  carefully  evaluated  to  ensure  appropriate  trust. 
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5.  Human  Performance  Issues  in  H-A  Teaming:  Operator’s  SA  of  the 
Tasking  Environment 


One  of  the  most  critical  factors  for  achieving  effective  supervisory  control  of  multiple  robots  is 
maintaining  adequate  SA  of  the  overall  tasking  environment  as  well  as  individual  robots.  There 
are  three  levels  of  SA  as  defined  by  Endsley  (1995): 

1 .  Perception  of  data  and  environmental  elements 

2.  Comprehension  of  the  current  situation 

3.  Projection  of  future  states  and  events 

Changes  in  the  environment  that  may  affect  the  plans  for  the  robots  need  to  be  detected  and  the 
plans  need  to  be  modified  in  a  timely  fashion.  Kaber  et  al.,  (2006)  evaluated  the  effectiveness  of 
adaptive  automation  (in  the  context  of  air  traffic  control)  for  supporting  different  stages  of 
information  processing  (i.e.,  information  acquisition,  information  analysis,  decision  making,  and 
action  implementation;  Parasuraman  et  al.,  2000).  They  reported  that  automating  the 
information  acquisition  part  of  the  task  was  most  beneficial  for  supporting  SA. 

Frequently,  changes  in  the  environment  may  require  the  operator  to  modify  his/her  plans  for  the 
robots.  According  to  the  National  Transportation  Safety  Board  (1994,  as  cited  in  Muthard  and 
Wickens,  2002),  nearly  two-thirds  of  aviation  accidents  caused  by  faulty  human  decision  making 
can  be  attributed  to  pilots’  failure  to  revise  their  flight  plans.  Muthard  and  Wickens  (2002) 
evaluated  the  effects  of  automation  on  pilots’  performance  in  plan  monitoring  and  revision. 

They  found  that  pilots  only  detected  about  30%  of  the  experimenter-induced  changes,  which 
should  have  resulted  in  flight  plan  revisions.  Plan  continuation  errors  were  especially 
pronounced  when  there  was  an  unreliable  automation  aid  as  compared  with  no  aid  present. 
Mumaw  et  al.,  (2001)  showed  an  even  more  alarming  inadequacy  in  monitoring  performance.  In 
their  study,  pilots  only  detected  about  3%  (1  of  32  total  cases)  of  unexpected  changes  to  the 
mode  of  an  automation  aid.  Indeed,  it  has  been  well  documented  that  operators  are  frequently 
unaware  of  mode  changes  when  interacting  with  automation  systems  and,  therefore,  are  confused 
about  the  systems’  behaviors  (Sarter  and  Woods,  1995).  In  fact,  data  show  that  even  if  changes 
in  the  environment  are  detected,  operators  may  have  difficulty  interpreting  the  relevance  of  the 
changes  and  their  effects  on  the  existing  plans  (Endsley,  1995). 

There  was  some  evidence  that  frequent  video  gamers  tended  to  have  better  SA  than  did 
infrequent  gamers  in  dynamic  replanning  tasking  environments  involving  H-A  teaming  for 
multi-robot  control  (Chen  and  Bames,  2012a;  Chen  and  Barnes,  2012b).  Given  findings  reported 
in  Green  and  Bavelier  (2006)  that  frequent  gamers  tend  to  have  better  visual  short-term  memory, 
it  is  not  surprising  to  find  them  exhibiting  better  SA  of  the  tasking  environments. 
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More  discussion  on  the  effects  of  gaming  experience  will  be  presented  in  the  section  on 
individual  differences.  The  rest  of  this  section  will  discuss  the  issues  related  to  multitasking  and 
task  switching,  both  of  which  have  significant  impact  on  the  operators’  development  and 
maintenance  of  SA,  as  well  as  potential  user  interface  solutions  to  address  the  SA  issues. 

5.1  Multitasking  and  Task  Switching 

On  the  future  battlefield,  Soldiers  will  very  likely  be  expected  to  perform  other  tasks 
concurrently  while  operating  a  robot  (or  multiple  robots)  (Chen  and  Barnes,  2012a).  While 
updates  of  information  are  needed,  recent  studies  have  also  shown  that  interrupting  a  primary 
task  (i.e.,  supervisory  control  of  robots)  with  an  intermittent  task  (e.g.,  communication  messages) 
can  have  a  negative  impact  on  SA  (Cummings,  2004;  Domeich  et  al.,  2012).  For  example, 
Cummings  (2004)  found  that  instant  messages  diverted  participants’  attention  from  their  primary 
task  (simulated  supervisory  control  of  Tomahawk  missiles),  thus  reducing  their  SA  when 
returning  their  focus  to  the  primary  task.  Therefore,  the  challenge  is  to  identify  the  means  by 
which  operators  can  most  effectively  maintain  SA. 

Researchers  have  studied  how  characteristics  of  the  automation  (e.g.,  reliability  and  autonomy) 
and  the  task  (e.g.,  complexity)  may  affect  performance  on  these  concurrent  tasks,  and  the 
supervisory  task  as  a  whole.  Manzey  et  al.,  (2012),  for  example,  investigated  performance 
consequences  of  automated  aids  (for  fault  identification  and  management)  that  are  occasionally 
unreliable  in  a  simulated  multitasking  supervisory  control  task.  The  results  showed  that 
automation  benefits  both  primary  (fault  diagnosis)  and  secondary  task  (response  to 
communications)  performance.  However,  a  significant  automation  bias  effect  was  observed.  In 
the  first  experiment,  about  half  of  the  participants  followed  a  wrong  diagnosis  generated  by  the 
automated  system  (versus  only  about  7%  of  participants  committed  the  same  error  manually);  in 
the  second  experiment,  about  20%  of  those  participants  who  had  not  experienced  automation 
unreliability  made  commission  errors  (versus  about  5%  of  those  who  had  experienced  system 
unreliability).  Furthermore,  in  both  experiments,  about  half  of  those  who  followed  the 
automation  wrongly  actually  made  an  effort  to  verify  the  appropriateness  of  the  diagnosis.  The 
authors  observed  that  the  data  seemed  to  suggest  individual’s  differences  in  susceptibility  to 
automation  bias  effects.  Section  6.1  will  discuss  individual  differences  in  attention  control 
ability  and  their  effects  on  human  interaction  with  automation.  Finally,  the  Manzey  et  al.,  (2012) 
study  reported  that,  with  the  highest  level  of  automation,  participants  showed  degraded  “return- 
to-manual”  performance.  The  authors,  therefore,  recommended  that  medium  levels  of 
automation  be  used  if  manual  skills  need  to  be  maintained. 

Simultaneous  control  of  multiple  robots  may  require  the  operator  to  switch  attention/control 
among  the  vehicles  from  time  to  time.  Basic  research  on  costs  of  task  switching  consistently 
shows  that  people’s  responses  tend  to  be  substantially  slower  and  more  error-prone  after  task 
switching  (Monsell,  2003;  Rubinstein  et  al.,  2001).  There  is  some  evidence  that  this  cost  may  be 
reduced  if  the  participants  have  a  chance  to  prepare  for  the  switch  or  receive  task  switching  cues 
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(Monsell,  2003;  Rubinstein  et  al.,  2001).  In  the  context  of  HRI,  research  has  been  conducted  to 
investigate  the  effects  of  task  switching  on  SA  and  operator  performance  (Crandall  et  al.,  2005; 
Parasuraman  et  al.,  2009;  Squire  et  al.,  2006;  Wang  and  Lewis,  2007).  SA  may  also  be  affected 
perceptually  as  a  result  of  the  change  blindness  phenomenon,  which  is  the  inability  to 
perceptually  attend  to  a  change  in  one’s  environment.  Parasuraman  et  al.,  (2009)  examined 
change  blindness  in  the  context  of  a  supervisory  control  task,  in  which  participants  were  asked  to 
monitor  a  UAV  and  a  UGV  video  feed  in  a  reconnaissance  tasking  environment.  Parasuraman 
et  al.,  (2009)  found  that  change  blindness  occurred  most  frequently  when  a  “distracter”  was 
present,  but  also  occurred  while  participants  shifted  their  attention  from  the  UAV  monitoring 
task  to  the  UGV  monitoring  task.  These  results  also  suggest  that  task  switching  during  a  robot 
supervisory  task  may  incur  change  blindness,  which  by  its  very  nature  affects  an  operator’s  SA 
(additional  information  on  task  switching  is  provided  in  the  next  paragraph).  According  to 
Norman’s  (1986)  seven  stages  of  user  activity,  interruptions  incur  the  greatest  cognitive  costs 
during  the  planning  phases  (intention  forming  and  action  planning)  as  well  as  the  evaluation 
phases  (outcome  interpretation  and  assessment).  Thus,  interface  designers  should  account  for 
this,  so  that  primary  tasks  are  only  interrupted  during  emergency  situations  or  during  moments  of 
low  workload  (e.g.,  after  evaluation  is  completed  or  before  initiating  a  new  plan).  However, 
alerts  should  be  provided  to  the  operator  indicating  the  changes  to  the  interface  and  the  degree  of 
importance  of  the  changes  (Sarter  et  al.,  2007). 

5.2  Implications  for  System  Design  -  Visualization  and  Attention  Management 
5.2.1  Visualization  Tools 

Proper  uses  of  information  visualization  techniques  can  help  the  operators  make  sense  of 
information  and,  thereby,  enhance  their  SA  of  their  tasking  environments  (for  a  review  on  human 
factors  issues  related  to  information  visualization,  see  Robertson  et  al.,  2009).  Dong  and  Hayes 
(2012)  showed  that  by  presenting  the  operators  simple  bar  graphs  to  indicate  uncertainty  aspects, 
the  operators  were  able  to  distinguish  ambiguous  and  unambiguous  choices  better  than  did  those 
without  access  to  the  visualization  display.  This  section  highlights  two  most-researched 
visualization  tools  in  the  context  of  HRI  and  human-automation  interaction,  augmented  reality 
and  ecological  interfaces. 

Augmented  Reality.  Augmented  reality  (also  known  as  synthetic  vision)  has  been  found  to  be 
an  effective  means  to  enhance  pilot/UAV  operator  SA  by  portraying  a  more  veridical  view  of  the 
combat  environment  (Calhoun  and  Draper,  2006).  For  example,  the  U.S.  Air  Force  has 
identified  several  candidate  synthetic  vision  overlay  concepts  for  UAV  applications  (Calhoun 
and  Draper,  2006).  The  following  information,  potentially,  can  be  overlaid  graphically  on  the 
streaming  video:  maps  and  other  synthetically  generated  symbology,  photo-imagery,  terrain 
elevation,  laser  range  scans,  past  and  potential  future  robot  paths,  updates  via  networked 
communication  with  other  sources,  cost  functions  related  to  plan  optimization,  and  other  vital 
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statistical  data  (Cummings  et  al.,  2012;  Calhoun  and  Draper,  2006;  Collett  and 
MacDonald,  2006).  However,  large  amounts  of  information,  while  helpful  in  reducing  the 
operator’s  scanning  effort  by  providing  more  data  in  a  centralized  area  (e.g.,  the  video),  can 
create  visual  clutter  and  degrade  operator’s  information  processing  (Wickens,  2005);  on  the  other 
hand,  Iani  and  Wickens  (2007)  suggested  that  the  attentional  tunneling  effect  of  the  augmented 
reality  displays  may  not  be  as  pronounced  as  previously  suggested.  Nevertheless,  it  is  beneficial 
that  a  declutter  capability  be  provided  so  the  operator  can  customize  the  overlaid  information 
presentation  according  to  the  situation  and  tasks  (Calhoun  et  al.,  2005). 

Ecological  Interface  Designs.  Ecological  interface  design  (EID)  is  a  user  interface  design 
technique  that  conveys  the  constraints  in  the  tasking  environment,  usually  visually  via  emergent 
patterns,  so  the  operator  can  intuitively  perceive  and  solve  the  problem  (Vincente,  2002; 

Vincente  and  Rasmussen,  1992).  Cummings  and  Bruni  (2010)  designed  a  user  interface  that 
supports  a  single  operator’s  ability  to  control  four  UAVs  simultaneously.  They  utilized  a 
configural  display  (which  conveys  system-level  information  by  portraying  emergent  visual 
features)  to  help  the  operator  visualize  the  overall  quality  of  plan  revisions.  In  another  study, 
Furukawa  and  Parasuraman  (2003)  demonstrated  that  EID  was  beneficial  for  enhancing  human 
operators’  detection  of  automation  errors  as  well  as  their  comprehension  of  system  states.  In 
their  first  experiment,  Furukawa  and  Parasuraman  (2003)  showed  that  human  operators,  using  an 
EID  display  showing  an  emergent  perceptual  feature,  were  able  to  detect  significantly  more 
system  errors  than  when  they  used  a  non-integrated  display  (i.e.,  they  showed  significantly  less 
automation-induced  complacency).  More  strikingly,  the  operators  were  able  to  achieve  better 
performance  even  though  their  visual  attention  to  the  EID  display  was  significantly  less, 
according  to  an  eye  movement  analysis,  indicating  that  their  monitoring  was  more  efficient.  In 
the  second  experiment,  Furukawa  and  Parasuraman  showed  the  effectiveness  of  an  EID  display 
that  portrays  graphically  the  intention  of  the  automated  system.  Their  results  showed  that  this 
visualization  tool  helped  the  human  operators  to  achieve  a  better  mental  model  of  the  system, 
which  enabled  them  to  make  better  decisions.  In  yet  another  study,  Furukawa  et  al.,  (2004) 
integrated  the  intention-represented  EID  display  in  a  partially-automated  process  control 
simulation  and  compared  its  effectiveness  with  that  of  an  EID  display  without  intention 
indicators  of  the  automated  system.  Results  showed  that  the  intention-represented  EID  display 
was  able  to  enhance  the  operators’  predictions  of  the  actions  and  behaviors  of  the  automated 
system  and,  therefore,  was  able  to  improve  the  operator’s  action  planning  and  decision  making. 
Additionally,  the  benefits  were  demonstrated  in  novel  scenarios,  suggesting  that  the  operators 
had  a  better  mental  model  of  the  automated  system  with  the  intention-represented  EID  display 
than  with  the  EID  display  without  the  intention  indicators. 
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5.2.2  Attention  Management  Tools  and  Interruption  Recovery  Aids 

When  controlling  multiple  robots  at  the  same  time,  it  is  inevitable  that  the  operator  will  focus  on 
some  aspects  of  the  environment  (e.g.,  one  of  the  robots)  before  resuming  his/her  monitoring  of 
all  the  robots.  Techniques  that  facilitate  task  resumption  have  been  proposed  and  tested  in 
various  tasking  environments  (Ratwani  et  al.,  2007;  Scott  et  al.,  2006;  St.  John  et  al.,  2005). 
Some  techniques  focus  on  reminding  the  operator  where  they  were  before  the  interruption 
(Ratwani  et  al.,  2007),  while  others  present  aids  for  the  operator  to  quickly  review  what 
happened  during  the  interruption  (Scott  et  al.,  2006;  St.  John  et  al.,  2005).  Ratwani  and  Trafton 
(2010)  recommended  that  in  interruption-prone  tasking  environments,  care  should  be  taken  to 
ensure  the  operator’s  visual  access  to  the  primary  task  in  order  to  facilitate  task  resumption  after 
the  interruptions.  Ratwani  et  al.,  (2007)  demonstrated  that  simply  by  reducing  the  size  (by  about 
75%)  of  the  window  for  the  interrupting  task  (i.e.,  reducing  the  occlusion  of  the  primary  task 
screen  by  the  interrupting  task  window),  participants  were  able  to  resume  their  primary  task 
significantly  faster.  Eye  tracking  data  also  showed  that  participants  were  more  accurate  at 
returning  to  where  they  left  off.  Other  more  sophisticated  techniques  to  facilitate  recovery  from 
interruptions  have  also  been  developed.  For  example,  St.  John  et  al.,  (2005)  discussed  the  utility 
of  a  SA  recovery  tool  (named  CHEX)  which  displayed  a  textual  event  history  list  in  a  naval  air 
warfare  environment  (monitoring  a  geoplot  of  an  airspace  which  contained  ownship  and 
approximately  50  other  aircrafts).  Other  researchers  have  proposed  replay  tools  that  can 
communicate  to  the  operators  recent  significant  events  via  video.  For  example,  Scott  et  al., 
(2006)  presented  two  types  of  replay  tools,  one  replaying  the  events  at  a  lOx  real  time  speed  and 
the  other  presenting  bookmarks  on  the  event  timelines  so  the  operator  could  view  the  replay  by 
selecting  the  bookmarks.  Results  showed  that  both  replay  techniques  were  effective,  especially 
when  the  tasking  environment  was  challenging.  Based  on  the  results,  the  authors  presented 
several  recommended  design  guidelines  for  interruption  assistance  interfaces: 

1.  Enable  user  control  of  event  replay. 

2.  Provide  visual  summary  of  critical  events  but  limit  the  summary  only  to  goal-related 
events 

3.  Clearly  indicate  relationships  between  past  and  current  system  state  (Scott  et  al.,  2006; 
p.  703) 

Domeich  et  al.,  (2012)  took  a  different  approach  and  developed  a  wearable  adaptive  system,  the 
Communications  Scheduler,  which  used  physiological  sensors  (EEG  and  electrocardiogram)  to 
detect  operators’  cognitive  workload  in  a  multitasking  situation  (including  navigation,  cognitive, 
monitoring,  and  maintaining  SA).  The  Communications  Scheduler  decided  whether  to  interrupt 
the  user’s  current  task  based  on  the  urgency  of  the  situation,  the  operator’s  cognitive  workload, 
and  the  system  state.  The  interruption  etiquette  was  designed  based  on  H-H  interactions. 
Empirical  testing  showed  that  the  Communication  Scheduler  positively  impacted  participants’ 
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performance  by  rescheduling  their  priorities,  resulting  in  only  a  temporary  loss  of  SA  for  low 
priority  messages. 


6.  Individual  Differences 


Multi-robot  control  inherently  involves  multitasking.  Multitasking  environments  often  require 
the  operator  to  switch  their  attention  among  the  tasks  effectively  (e.g.,  receiving  and  processing 
intelligence  data,  sending  spot-reports,  maintaining  communications  with  fellow  crew  members, 
etc.),  with  or  without  agents’  help.  Manzey  et  al.,  (2012)  observed  significant  individual 
differences  in  susceptibility  to  automation  bias  effects  in  the  multitasking  environments  they 
simulated,  although  the  authors  did  not  identify  what  individual  differences  factors  contributed  to 
the  observed  behaviors.  Previous  research  has  shown  that  some  individuals  show  more 
performance  decrements  when  multitasking  than  others  and  these  decrements  may  be  related  to 
their  poorer  abilities  to  control  and  allocate  attention  (Feldman  Barrett  et  al.,  2004;  Rubinstein 
et  al.,  2001;  Schumacher  et  al.,  2001).  These  results  suggest  that  individual  differences  in 
attentional  control  seem  to  play  a  critical  role  in  determining  an  operator’s  overall  multitasking 
performance.  Research  also  shows  that  individual  differences  in  spatial  ability  and  gaming 
experience  play  important  roles  in  determining  operators’  SA  in  multi-robot  tasking 
environments  (Chen  and  Barnes,  2012a;  Chen  and  Barnes,  2012b).  The  following  section  briefly 
reviews  these  individual  differences  factors  that  may  impact  the  overall  effectiveness  of  H-A 
teaming  for  multi-robot  control. 

6.1  Attentional  Control 

Attentional  control  is  defined  as  one’s  ability  to  focus  and  shift  attention  in  a  flexible  manner 
(Derryberry  and  Reed,  2002).  According  to  a  recent  U.S.  Air  Force’s  survey  of  subject  matter 
experts  on  the  performance  of  UAV  operators  (Chapelle  et  al.,  2010),  attentional  control  is  one 
of  the  most  important  abilities  that  affect  an  operator’s  performance  since  the  robotics  control 
task  is  inherently  multitasking  (e.g.,  sensor  manipulation,  tracking,  communication,  etc.).  Past 
research  has  shown  that  poor  attention  allocation  was  related  to  degraded  human  performance  in 
multi-robot  control  (Crandall  and  Cummings,  2007;  Goodrich  et  al.,  2005).  Several  studies  have 
shown  that  there  are  individual  differences  in  multitasking  performance,  and  some  people  are 
less  prone  to  performance  degradation  during  multitasking  conditions  (Rubinstein  et  al.,  2001; 
Schumacher  et  al.,  2001).  There  is  evidence  that  people  with  better  attentional  control  can 
allocate  their  attention  more  flexibly  and  effectively,  and  attention-switching  flexibility  can 
predict  performance  of  such  diverse  tasks  as  flight  training  and  driving  (Bleckley  et  al.,  2003; 
Derryberry  and  Reed,  2002;  Feldman  Barrett  et  al.,  2004;  Kahneman  et  al.,  1973). 
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Several  studies  showed  that  operators  with  lower  attentional  control  interacted  differently  with 
automated  systems  than  did  those  with  higher  attentional  control;  they  tended  to  rely  more 
heavily  on  automation,  even  when  the  reliability  of  those  systems  was  low  (Chen  and  Terrence, 
2009;  Chen  and  Barnes,  2012a;  Thropp,  2006).  For  example,  it  appears  that  for  high  attentional 
control  participants,  false-alarm-prone  systems  tend  to  be  more  detrimental  than  miss-prone 
alerts,  due  to  “disuse”  of  automation  (although  this  difference  is  moderated  by  the  ease  of 
verification  of  the  alerts’  validity,  as  shown  in  Chen  and  Barnes,  2012a).  For  low  attentional 
control  participants,  conversely,  miss-prone  automation  was  more  harmful  than  false-alarm- 
prone  automation,  due  to  “misuse”  of  (i.e.,  over-reliance  on)  automation.  According  to  Feldman 
Barrett  et  al.,  (2004),  those  with  lower  attentional  control  tend  to  take  the  “cognitive  miser” 
approach  (i.e.,  conserving  the  amount  of  cognitive  resources)  when  dealing  with  complex 
information  processing  in  order  to  reduce  the  attentional  control  requirements.  When  dealing 
with  automated  systems,  therefore,  it  is  likely  that  operators  with  different  levels  of  attention 
switching  abilities  may  react  differently  to  unreliable  automated  systems.  In  other  words, 
operators’  behaviors  of  compliance  with,  and  reliance  on,  automation  may  be  moderated  based 
on  their  ability  to  effectively  switch  their  attention  among  the  systems.  For  example,  the  well 
documented  automation-induced  complacency  effects  (Dzindolet  et  al.,  2001;  Parasuraman  and 
Manzey,  2010;  Thomas  and  Wickens,  2004;  Young  and  Stanton,  2007)  may  be  more  severe  for 
poor  attentional  control  individuals  compared  with  those  with  better  attentional  control.  This 
phenomenon  has  been  demonstrated  in  Chen  and  Terrence  (2009)  and  Chen  and  Bames  (2012a). 

Their  results  are  consistent  with  past  research  that  self-confidence  is  a  critical  factor  in  mediating 
the  effect  of  trust  on  automation  reliance  (Lee  and  Moray,  1992;  Lee  and  See,  2004).  These 
findings  have  implications  for  designs  of  agents  for  multi-robot  control.  More  research  should 
also  investigate  training  interventions  (e.g.,  attention  management)  and/or  user  interface  designs 
(e.g.,  multimodal  cueing  displays,  automation  transparency,  and  visualization  techniques)  that 
can  mitigate  performance  shortfalls  of  those  with  lower  attentional  control  (Chen  et  al.,  2007; 
Chen  et  al.,  2011;  Dux  et  al.,  2009). 

6.2  Spatial  Ability 

Spatial  ability  has  been  found  to  be  a  significant  factor  in  certain  visual  display  domains 
(Stanney  and  Salvendy,  1995),  multitasking  involving  flight  asset  monitoring  and  management 
(Morgan  et  al.,  2011),  virtual  environment  navigation  (Chen  et  al.,  2000),  visual  search  task 
(Chen,  2010;  Chen  and  Barnes,  2012a;  Chen  et  al.,  2008;  Chen  and  Joyner,  2009;  Chen  and 
Terrence,  2008,  2009;  Fincannon  et  al.,  2012),  and  robotics  task  performance  (Cassenti  et  al., 
2009;  Lathan  and  Tracey  2002).  U.S.  Air  Force  scientists  (Chappelle  et  al.,  2010;  Chappelle 
et  al.,  2010)  interviewed  53  subject  matter  experts  about  abilities  that  were  critical  to  effective 
performance  of  UAV  control  tasks  in  terms  of  piloting  and  sensor  operations — spatial  ability  was 
identified  as  an  important  factor  for  both  tasks.  Our  previous  research  showed  that  individuals 
with  higher  spatial  ability  exhibited  more  effective  visual  scanning  and  target  detection 
performance  (Chen,  2010;  Chen  and  Bames,  2012a;  Chen  and  Bames,  2012b;  Chen  et  al.,  2008; 
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Chen  and  Joyner,  2009;  Chen  and  Terrence,  2008,  2009).  Based  on  these  findings,  it  seems 
reasonable  to  expect  operators  with  higher  spatial  ability  to  perform  better  with  agents  in  multi¬ 
robot  control  tasking  environments,  especially  if  more  spatial  information  processing  is  required 
from  the  human.  Alternatively,  training  interventions  that  could  enhance  the  spatial 
interpretations  required  to  perform  a  mission  task  might  also  be  of  benefit  (Baldwin  and  Reagan, 
2009). 

6.3  Gaming  Experience 

According  to  Green  and  Bavelier  (2006)  and  Hubert- Wallander  et  al.,  (2010),  experienced  action 
video  game  players,  compared  with  infrequent/non-  gamers,  were  found  to  perform  significantly 
better  on  tasks  that  required  visuo-spatial  selective  attention,  multiple  object  tracking,  rapid 
processing  of  visual  information  and  imagery,  and  flexibility  in  attention  allocation.  Hambrick 
et  al.,  (2010)  also  demonstrated  the  relationship  between  video  game  experience  and 
multitasking  performance.  A  U.S.  Air  Force  study  (Triplett,  2008)  concluded  that,  based  on 
interviews  of  UAV  pilots,  gamers’  superior  visual  information  processing  skills  may  be  able  to 
translate  into  superior  robotics  management  performance.  Indeed,  a  recent  U.S.  Air  Force  study 
(McKinley  et  al.,  2010)  found  that  frequent  video  gamers  outperformed  infrequent  gamers  on 
robotics  (UAV)  tasks  and,  in  some  cases,  performed  as  well  as  experienced  pilots.  Additionally, 
Cummings  et  al.,  (2010)  found  that  frequent  gamers  collaborated  more  with  an  automated  UAV 
replanning  system  (higher  degree  of  consent)  than  did  infrequent  gamers.  Finally,  Chen  and 
Bames  (2012a)  and  Chen  and  Bames  (2012b)  demonstrated  that  frequent  gamers  exhibited 
significantly  better  SA  of  the  tasking  environments  than  did  infrequent  gamers.  Based  on  these 
findings,  therefore,  it  is  expected  that  frequent  gamers  may  work  better  with  agents  in  multi¬ 
robot  control  tasking  environments  due  to  their  superiority  in  visual  attention  allocation  and 
information  processing. 


7.  Agent  as  Personal  Assistants  for  Supervisory  Control  of  Multiple 
Intelligent  Systems 


Most  agent  technologies  that  we  reviewed  have  the  same  drawbacks;  that  is,  they  require  focused 
attention,  have  high  workload  demands,  and  cause  diffusion  of  SA  when  multiple  agents  are 
involved.  The  problem  becomes  more  acute  as  the  number  of  intelligent  systems  increases  and 
the  human  operator  is  expected  to  multitask  (Barnes  and  Evans,  2010;  Chen  and  Bames,  2012a). 
RoboLeader  is  closer  to  the  concept  of  a  personal  assistant  agent  (e.g.,  the  Cognitive  Assistant 
that  Learns  and  Organizes  (CALO)  project,  see  Yorke-Smith  et  al.,  2012)  rather  than  a 
specialized  robotic  entity  (Barnes  and  Grossman,  1985;  Jones  and  Schmidlin,  2011).  The 
principle  advantage  of  the  personal  assistant  concept  is  that  the  agent  interfaces  directly  with  the 
operator  in  order  to  coordinate  other  intelligent  systems — freeing  the  operator  for  other  tasks 
such  as  for  scanning  other  information  sources.  The  agent  is  also  an  interface  conveying 
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information  from  the  subordinate  intelligent  systems  to  the  human  operator  and  vice  versa.  The 
final  version  of  an  intelligent  agent  as  a  personal  assistant  needs  to  be  refined  and  some  of  our 
preliminary  assumptions  need  to  be  tested.  Also,  some  of  the  guidelines  are  not  currently 
feasible  and  depend  on  future  research  in  such  areas  as  machine  learning,  natural  language,  and 
cognitive  architectures  (Kelley,  2006).  Based  on  the  literature  and  RoboLeader  research,  the 
following  are  our  initial  “best  guess”  guidelines  for  an  agent  as  a  personal  assistant  in  non¬ 
routine  environments  especially  when  safety  is  involved  (e.g.,  combat): 

a.  Agent/human  interaction  needs  to  be  flexible.  The  system  should  be  able  to 
adjust  to  operator  workload  and  to  allow  agents  to  act  autonomously  under 
specified  conditions  (Goodrich,  2010;  Tecuci  et  al.,  2007). 

b.  Operators  must  have  ultimate  decision  authority,  The  mechanism  for  ensuring 
human  authority  needs  to  be  embedded  in  the  agent  architecture  (e.g.,  mixed 
initiative  systems);  however,  if  human  supervisors  are  “out  of  the  loop”  too  long, 
complacency  effects  will  occur  (Endsley  and  Kiris,  1995;  Fem  and  Shively,  2009; 
Parasuraman  and  Riley,  1997;  Parasuraman  and  Manzey,  2010;  Woods  et  al., 

2004). 

c.  Humans  and  agents  have  different  cognitive  characteristics.  Mathematical  or 
algorithmic  solutions  for  well  defined  problem  spaces  are  best  handled  by  agents; 
uncertain  situations  that  are  poorly  defined  or  problems  that  have  meta¬ 
consequences  are  best  handled  by  experienced  humans  (Bradshaw  et  al.,  2011; 
Kahneman  and  Klein,  2009). 

d.  Automation  transparency  is  essential.  Bi-directional  communications  between 
the  operator  and  the  agent  need  to  be  facilitated  so  the  operator  can  understand  the 
agent’s  intent  and  vice-versa.  Intent  understanding  and  communication  with  agents 
is  limited  but  is  currently  feasible  for  specific  task  environments.  A  personal 
agent’s  cognitive  architecture  should  be  designed  to  be  congruent  with  human 
cognition  (Avery  et  al.,  2006;  Chen  and  Bames,  2012a;  Jones  and  Schmidlin,  2011; 
Tellex  et  al,  2011). 

e.  Human  individual  differences  must  be  part  of  the  human/agent  design  process. 

This  can  be  accomplished  either  by  selection,  training  or  even  designing  agents  that 
are  sensitive  individual  differences  among  humans  (Chen  and  Barnes,  2012a;  Chen 
and  Bames,  2012b). 
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8.  Conclusions 


We  have  reviewed  a  number  of  specialized  examples  of  agent  technology  for  multi-robot  control 
and  human  performance  issues  that  need  to  be  taken  into  account  when  designing  such  systems. 
The  proliferation  of  agent  technology  will  likely  keep  progressing  and  be  used  increasingly  in 
more  complex  environments — both  because  of  improvements  in  computing  power  and  the 
development  of  more  sophisticated  algorithms.  Artificial  agents  are  able  to  supplement  human 
intelligence  by  their  use  of  more  formal  logic  and  their  use  of  complex  optimization  algorithms 
to  solve  circumscribed  problem  sets.  Human  cognitive  strengths,  in  turn,  can  be  complementary 
to  artificial  intelligence  capabilities.  In  the  context  of  military  operations,  agents  can  reduce 
Soldier  workload  by  being  able  to  attend  to  multiple  functions  that  might  overwhelm  Soldiers 
during  the  heat  of  combat.  However,  no  set  of  agents  in  the  foreseeable  future  will  be  able  to 
understand,  for  example,  the  nuanced  political  and  ethical  implications  facing  U.S.  ground  troops 
(Linegang  et  al.,  2006).  Therefore,  for  most  situations,  complete  agent  autonomy  is  not 
desirable,  especially  in  the  near  or  mid-term.  Rather,  H-A  teams  and  mixed-initiative  tasking 
have  the  greatest  potentials  of  supporting  fluid  interactions  between  humans  and  agents  while 
taking  advantage  of  the  strengths  of  both.  Agents  can  be  delegated  to  perform  specific  task 
autonomously  but  only  with  the  permission  of  the  human  operator  and,  if  possible,  only  if  the 
human  is  able  to  intervene  when  necessary.  For  future  H-A  teams,  a  crucial  link  will  be  implicit 
and  explicit  communication  between  humans  and  agents.  NLP  by  itself  is  insufficient;  some 
ability  to  have  a  common  cognitive  understanding  and  for  both  the  human  and  robot  to 
communicate  their  intentions  will  be  necessary  (Chen  et  al.,  201 1).  Also  as  pointed  out  by 
Lee  (2008)  in  his  review  of  the  seminal  article  on  human-automation  interaction  by  Parasuraman 
and  Riley  (1997),  “automation  requires  more,  not  less,  attention  to  training,  interface  design,  and 
interaction  design”  (Lee,  2008,  p.  404).  As  suggested  by  Parasuraman  and  Manzey  (2010), 
automation  does  not  replace  the  human,  rather,  it  changes  the  nature  of  the  human’s  task. 

The  missing  element  in  current  research  is  the  development  of  a  true  partnership  between 
humans  and  artificially  intelligent  entities.  Work  cited  suggests  the  underpinning  of  partnership 
relationship  in  trust,  artificial  animal  surrogates,  advanced  interfaces,  collaborative  behaviors, 
supervisory  control  allocated  to  robotic  entities,  and  even  shared  ethical  values.  Whereas  these 
ongoing  research  investigations  all  contribute  to  the  advancements  in  H-A  teaming  capabilities, 
one  of  the  most  important  elements  of  true  H-A  partnership,  shared  linguistic  awareness,  is  still 
in  a  nascent  state  (Tellex  et  al.,  201 1).  Natural  language  interfaces  by  themselves  are  not 
sufficient  for  peer  relationships  which  require  more  robust  shared  cognitive  architectures  that 
allow  the  artificial  entity  to  infer  intent,  generalize  instructions,  and  participate  in  a  problem 
solving  dialogue  in  novel  situations.  Intelligent  agents  in  a  mature  state  that  can  operate  as 
partners  in  truly  complex  environments  will  most  likely  not  be  available  in  the  short  term. 
However,  continuing  research  to  understand  the  characteristics  of  partnership  relationships  is  an 
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important  requisite  for  understanding  the  boundaries  of  the  relationship  (Chen  and  Barnes, 
2012a;  Chen  and  Barnes,  2012b).  For  example,  delineating  H-H  problem  solving  in 
environments  of  interest  will  help  to  isolate  cognitive  factors  necessary  to  communicate  in  novel 
environments.  Investigating  cognitive  architectures  in  simpler  environments  will  help  measure 
what  variables  influence  peer  synergy  for  intent  inferencing  and  two-way  dialogues.  There  is  no 
requirement  for  intellectual  equality  between  human  and  artificial  agents;  the  important  issue  is 
to  understand  what  factors  are  necessary  for  partnership  problem  solving  in  truly  complex  real 
world  environments. 
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List  of  Symbols,  Abbreviations,  and  Acronyms 


3P’s 

purpose,  process,  and  performance 

ACT-R 

Adaptive  Control  of  Thought-Rational 

AIA 

Adaptive  Intelligent  Agent 

CAST 

Collaborative  Agents  for  Simulating  Teamwork 

CALO 

Cognitive  Assistant  that  Learns  and  Organizes 

CCAS 

Cognitive  and  Cooperative  Assistant  System 

EEG 

electroencephalography 

EID 

Ecological  interface  design 

FAP 

false  alarm-prone 

H-A 

human-agent 

H-H 

human-human 

HRI 

human-robot  interaction 

IED 

improvised  explosive  device 

LOA 

levels  of  autonomy 

MAPGEN 

Mixed-initiative  Activity  Plan  GENerator 

MP 

miss -prone 

NASA 

National  Aeronautics  and  Space  Administration 

NLP 

natural  language  processing 

R-CAST 

Recognition-Primed  Decision  (RPD)  framework 

RPD 

Recognition  Primed  Decision 

SA 

situation  awareness 

SOAR 

Symbolic  Cognitive  Architecture 
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SS-RICS 


Symbolic  and  Sub-symbolic  Robotic  Intelligence  Control  System 


UAV 

UGV 


unmanned  aerial  vehicle 
unmanned  ground  vehicle 
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NO.  OF 

COPIES  ORGANIZATION 


NO.  OF 

COPIES  ORGANIZATION 


1  DEFENSE  TECHNICAL 
(PDF  INFORMATION  CTR 
only)  DTIC  OCA 

8725  JOHN  J  KINGMAN  RD 
STE  0944 

FORT  BELVOIR  VA  22060-6218 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
IMAL  HRA 

2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIO  LL 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  C  A  DAVISON 

320  MANSCEN  LOOP  STE  1 15 
FORT  LEONARD  WOOD  MO  65473 

2  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DI 

T  DAVIS 

J  HANSBERGER 

BLDG  5400  RM  C242 

REDSTONE  ARSENAL  AL  35898-7290 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRS  EA  DR  V  J  RICE 
BLDG  4011  RM  217 
1750  GREELEY  RD 

FORT  SAM  HOUSTON  TX  78234-5002 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DG  K  GUNN 
BLDG  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
ARMC  FIELD  ELEMENT 
RDRL  HRM  CH  C  BURNS 
THIRD  AVE  BLDG  1467B  RM  336 
FORT  KNOX  KY  40121 

1  ARMY  RSCH  LABORATORY  -  HRED 
AWC  FIELD  ELEMENT 
RDRL  HRM  DJ  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FORT  RUCKER  AL  36362-5000 


1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CK  J  REINHART 
10125  KINGMAN  RD  BLDG  317 
FORT  BELVOIR  VA  22060-5828 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AY  M  BARNES 
2520  HEALY  AVE 
STE  1172  BLDG  51005 
FORT  HUACHUCA  AZ  85613-7069 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HR  MP  D  UNGV ARSKY 
POPE  HALL  BLDG  470 
BCBL  806  HARRISON  DR 
FORT  LEAVENWORTH  KS  66027-2302 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AT  J  CHEN 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AT  C  KORTENHAUS 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CU 
6501  E  11  MILE  RD  MS  284 
BLDG200A  2ND  FL  RM2104 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
FIRES  CTR  OF  EXCELLENCE 
FIELD  ELEMENT 
RDRL  HRM  AF  C  HERNANDEZ 
3040  NW  AUSTIN  RD  RM  221 
FORT  SILL  OK  73503-9043 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AV  S  MIDDLEBROOKS 
91012  STATION  AVE 
FORT  HOOD  TX  76544-5073 

1  ARMY  RSCH  LABORATORY  -  HRED 
HUMAN  RSRCH  AND  ENGRNG 
DIRCTRT  MCOE  FIELD  ELEMENT 
RDRL  HRM  DW  E  REDDEN 
6450  WAY  ST 
BLDG  2839  RM  310 
FORT  BENNING  GA  31905-5400 
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NO.  OF 

COPIES  ORGANIZATION 


1  ARMY  G1 
(CD  DAPE  MR  B  KNAPP 
only)  300  ARMY  PENTAGON  RM  2C489 
WASHINGTON  DC  20310-0300 


ABERDEEN  PROVING  GROUND 

3  DIR  USARL 
RDRL  HR 
L  ALLENDER 
RDRL  HRM 

P  S  A V AGE-KNEPS  HIELD 
RDRL  HRS  D 
B  AMREIN 
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